CubeFS in Action: Empowering Users Through Case Studies - Leon Chang, OPPO

CubeFS in Action: Empowering Users Through Case Studies

Speakers: Leon Chang, Maintainer of CubeFS, OPPO

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=_rDE1PD5Z5I

Overview

This talk, presented by Leon Chang from OPPO, a maintainer of the open-source project CubeFS, offers a comprehensive exploration of CubeFS as a next-generation cloud-native storage system. The presentation outlines its robust architecture, highlights key features, and delves into several real-world end-user case studies, primarily focusing on its application in artificial intelligence (AI) workloads and compute-storage separation scenarios. Chang, who also oversees OPPO's file system, details how CubeFS addresses critical challenges in modern data management, including scalability, performance, cost optimization, and data consistency in complex, distributed environments.

The significance of this talk lies in its demonstration of CubeFS's practical utility in demanding enterprise settings, particularly within the rapidly evolving AI landscape. As AI models grow in complexity and data volumes skyrocket, efficient and cost-effective storage solutions become paramount. CubeFS provides a compelling answer to these challenges, showcasing its ability to support high-throughput training, rapid model distribution, and intelligent data tiering. Furthermore, its application in separating compute and storage for systems like ClickHouse underscores its versatility in optimizing resource utilization and enhancing system reliability across diverse cloud-native architectures.

Chang's presentation, his first English speech, serves as a testament to CubeFS's maturity and its recent graduation from the CNCF. It provides invaluable insights for organizations grappling with large-scale data storage, particularly those heavily invested in AI development or seeking to modernize their data infrastructure with cloud-native principles. By presenting concrete case studies from OPPO's experience, the talk offers a practical roadmap for leveraging CubeFS to empower users and drive innovation in data-intensive applications.

Background

▶ Watch: Introduction, CubeFS overview, and CNCF graduation (0:00)

The journey towards cloud-native architectures has fundamentally reshaped the requirements for storage systems. Traditional monolithic storage solutions often struggle with the dynamic scaling, high performance demands, and cost efficiencies expected in modern cloud environments. This context gave rise to projects like CubeFS, which joined the CNCF in 2019 and, after approximately five years of development and a dozen versions, proudly graduated from the foundation at the end of last year. CubeFS positions itself as an independent, self-governing, next-generation cloud-native storage system designed to meet these evolving needs.

At its core, CubeFS is engineered with a modular and highly scalable architecture. The client subsystem is a complex component offering multiple interfaces, including S3, HDFS, and POSIX, enabling broad compatibility with existing applications. It frequently interacts with the backend, managing the entire data flow process. A critical innovation is the cache subsystem, specifically designed for acceleration scenarios, becoming increasingly vital for AI workloads. The metadata subsystem is meticulously crafted for strong consistency and easy scalability, supporting essential system-level functions like trash management, auditing, and autonomic POSIX interfaces. For object access, CubeFS employs an object access zone that differs from traditional object storage by building capabilities atop its file system engine, essentially acting as a proxy. The storage subsystem, the data layer, includes two distinct engines: a multi-replica engine for high availability and a erasure coding (EC) engine for cost-effective, high-density storage, each with its own metadata management.

Key features underpinning CubeFS’s capabilities include its support for multiple protocols, the dual storage engines (multi-replica and erasure coding), and the achievement of strong consistency across all main components based on Raft and Quorum algorithms. Its distributed cache capability is a cornerstone for acceleration, particularly in AI. Furthermore, intelligent tiering helps optimize costs by migrating data from high-performance to lower-cost media. Finally, CubeFS offers seamless integration with Kubernetes through its CSI plugin, allowing it to be easily deployed and managed within containerized environments. These architectural choices and features directly address common pain points in large-scale data management, particularly those amplified by the demands of artificial intelligence and distributed computing.

Key Findings

▶ Watch: Key features: multi-protocol, dual engine, consistency, caching (4:00)

The presentation highlights several significant findings and contributions of CubeFS, demonstrated through its application in various real-world scenarios at OPPO:

• Optimized AI Storage Solution: CubeFS provides a comprehensive storage solution tailored for the entire AI lifecycle—data processing, model training, and inference. It addresses the challenges of large datasets, high-throughput requirements during training, and rapid model distribution, offering capabilities like handling diverse data sources, massive scalability, and low-latency access.
• Effective Hybrid Cloud AI Training: For organizations like OPPO that leverage public cloud resources for compute-intensive AI training, CubeFS facilitates a cost-effective hybrid cloud strategy. By combining self-built cloud storage for full data persistence with public cloud caching for hot data, it significantly reduces transfer costs and improves the performance of repeated reads, mitigating the challenges of geographic data distribution.
• Advanced Distributed Model Distribution: The distributed cache system within CubeFS is a critical enabler for efficient model distribution across multiple regions and inference endpoints. It features prompt distribution through path-based synchronization, robust consistency control, and pre-warming capabilities, ensuring models are available quickly and reliably where needed.
• Intelligent Data Tiering for Cost Efficiency: CubeFS successfully implements intelligent data tiering, a crucial capability for managing the escalating costs of storing massive AI datasets. By automatically migrating "cold" data (e.g., 80% of AI data not read within three months) from expensive high-performance media (like NVMe) to lower-cost storage, while retaining "hot" data (<10% of total) on high-performance tiers, it achieves a significant balance between computational performance and budget constraints.
• Enhanced Compute-Storage Separation for Databases: CubeFS demonstrates its value in achieving robust compute-storage separation, specifically for analytical databases like ClickHouse. Through an architectural evolution from initial single-replica volumes to a fully shared storage model, CubeFS provides inherent reliability, self-repair capabilities for disk failures, and simplifies operational maintenance, moving the burden of data consistency and repair away from the application layer.
• High-Performance SDK-based Access: For applications with extreme performance and stability requirements, CubeFS offers an SDK-based client that bypasses the overheads and constraints of traditional FUSE-based access. This user-mode SDK delivers significantly higher throughput and lower latency, making it ideal for performance-critical use cases such as key-value stores (e.g., RocksDB, Redis) and append-only write applications.

Technical Deep Dive

▶ Watch: First use case: Building AI storage with CubeFS (4:40)

CubeFS's technical prowess is best understood by dissecting its architectural components and how they are leveraged to solve complex storage challenges.

The foundational CubeFS architecture is designed for cloud-native environments. The client subsystem is versatile, supporting S3, HDFS, and POSIX interfaces, making it adaptable to a wide array of applications without requiring significant modifications. This client-side intelligence frequently communicates with the backend, orchestrating data access. The cache subsystem is a new design becoming increasingly important for AI acceleration. The metadata subsystem employs strong consistency, achieved through Raft consensus, ensuring data integrity and easy scalability. It underpins crucial features like trash management, auditing, and POSIX semantics. The object access zone integrates object storage capabilities directly with the file system engine, providing a unique approach to data access. Finally, the storage subsystem offers two distinct data engines: a multi-replica engine for high availability and an erasure coding (EC) engine for efficient, cost-optimized storage. Both engines manage their own metadata and include inspection and management systems. Strong consistency, based on Raft and Quorum, is a hallmark across all main components.

In the context of AI storage, CubeFS addresses the entire AI workflow:

• Data Processing: CubeFS handles diverse data sources and provides massive scalability to accommodate the tremendous amounts of data generated, filtered, and cleaned during this stage, reducing the need for data movement between different storage systems.
• Model Training: For scenarios like computer vision, where data reading is repetitive, CubeFS delivers high throughput and low latency. While large language models (LLMs) might load all data at once, other training paradigms heavily benefit from optimized read performance.
• Inference: The system ensures high throughput for rapid model distribution to endpoints, a critical requirement for quick deployment of trained models.

Addressing the challenges of large language models (LLMs) and the ever-growing demand for computational resources, OPPO leverages CubeFS in a hybrid cloud AI training strategy. The core idea is to store the full dataset in a self-built private cloud, while "hot data" (frequently accessed data) is cached in public cloud environments closer to compute resources. This approach significantly reduces data transfer costs associated with moving entire datasets across regions, especially when compute resources are provisioned elastically on public clouds. Although the initial read latency might be higher, subsequent reads of cached data exhibit much better performance. Consistency is maintained by ensuring incremental updates are synchronized, reducing the need for full data transfers.

The distributed cache system is pivotal for efficient model distribution, especially across geographically dispersed regions. It requires several key features:

• Prompt Distribution: Through path configuration, files written within a specified path can be synchronized to distributed cache endpoints, enabling "pre-warming" during the write process itself.
• Consistency Control: A robust control system prevents clients from accessing stale data.
• Pre-warming Capability: This is a service-driven feature where specific directories are designated as tasks for pre-warming, driven by a measurement system, integrating seamlessly with business operations.

Technically, the distributed cache system prioritizes high performance, high throughput, and load balancing. It utilizes consistent hashing to distribute files. Each file is divided into 1MB segments, and each segment is mapped to a slot partition. Multiple slot partitions are then mapped to a single cache node, defining the routing rule. This system supports both memory and disk storage. Benchmark tests indicate that each cache node with 4TB of disk capacity can provide 50 GB network throughput, with the network bandwidth being the primary bottleneck. The design also incorporates elastic cache replicas, configurable based on business needs (e.g., single copy for regular operations, multi-replica for massive requests to hot data or global distribution). Distance awareness allows clients to select the replica cache node with the lowest network latency. CubeFS is also actively seeking cooperation with communities like Fluid to provide integrated backend persistent storage and cloud data cache acceleration.

Intelligent data tiering is crucial for cost optimization. OPPO's analysis revealed that for most AI businesses, 80% of data is not read within three months, and "hot data" accounts for less than 10% of the total. Storing 10 petabytes of data on NVMe disks incurs significant annual costs. CubeFS addresses this with a lifecycle system fulfilled by its LC node module. This system runs tasks to migrate cold data to lower-cost storage media. During migration, client leases are used to ensure normal read and write operations are unaffected. For data security, the original data is retained for a period after transfer, with checks on both sides to guarantee consistency.

For compute-storage separation in ClickHouse, CubeFS provides a more robust solution than traditional direct-attached storage. Initially, ClickHouse used two CubeFS clusters with single-replica volumes, achieving unlimited storage and cost reduction but lacking storage-level consistency and automatic repair. The updated architecture positions CubeFS as a shared storage solution. In this model, ClickHouse fully depends on CubeFS, which inherently provides reliability and self-repair capabilities for disk failures through its multi-replica engine. This significantly improves stability and simplifies operations and maintenance, making disk failures transparent to the business.

Finally, for extreme performance requirements, CubeFS offers an SDK-based client as an alternative to the standard FUSE client. FUSE, while versatile, can introduce performance overhead and impose constraints like boot block size. The SDK runs entirely in user mode, bypassing the kernel mode, which results in significantly higher throughput, performance, and stability. This is particularly beneficial for applications like key-value stores (e.g., RocksDB, Redis) that require massive, stable storage with stringent P99 latency requirements (within one millisecond), or for append-only write applications storing WAL logs and SST files.

Demo / Proof of Concept

▶ Watch: Addressing storage challenges for Large Language Models (8:50)

While the talk "CubeFS in Action: Empowering Users Through Case Studies" did not feature a live technical demonstration or a specific proof-of-concept walkthrough, it effectively served a similar purpose by presenting detailed architectural diagrams, performance metrics, and real-world implementation case studies from OPPO. These case studies, particularly those concerning AI storage, distributed caching, intelligent tiering, and ClickHouse integration, functioned as practical demonstrations of CubeFS's capabilities and its efficacy in solving complex enterprise storage challenges. The speaker focused on how CubeFS’s features translate into tangible benefits and operational improvements for end-users, rather than showcasing a real-time system interaction.

Defensive Implications

▶ Watch: Optimizing AI computational resource consumption strategies (10:00)

For organizations leveraging or considering CubeFS, understanding its capabilities and design principles offers several crucial defensive implications:

• Cost Optimization through Intelligent Tiering: Defenders should actively implement and configure CubeFS's intelligent data tiering (LC node module) to migrate infrequently accessed data (cold data) to lower-cost storage media. This is a primary defense against escalating storage costs, especially with large datasets typical of AI/ML workloads. Regular analysis of data access patterns can inform optimal tiering policies.
• Enhanced AI Workflow Performance and Reliability: By adopting CubeFS for AI workloads, organizations can ensure high-throughput and low-latency storage for data processing, training, and inference. The distributed cache system, with its pre-warming capabilities and consistency controls, directly contributes to faster model distribution and more reliable AI pipelines, reducing operational risks associated with slow or inconsistent data access.
• Robust Compute-Storage Separation for Critical Databases: For analytical databases like ClickHouse, migrating to a shared CubeFS storage model (as opposed to single-replica volumes) provides inherent reliability and self-healing capabilities for disk failures. This defends against data loss and service disruption, significantly simplifying disaster recovery and operational maintenance. Defenders should prioritize this shared storage architecture for critical data platforms.
• Leveraging SDK for Performance-Critical Applications: For applications demanding extreme performance and low latency, such as key-value stores or append-only write systems, utilizing the CubeFS SDK client (which bypasses FUSE overhead) is a defensive strategy to ensure stability and meet strict SLA requirements (e.g., P99 latency within 1ms). This reduces the risk of performance bottlenecks impacting critical online services.
• Strong Consistency Guarantees: CubeFS's use of Raft and Quorum for strong consistency across metadata and multi-replica data provides a strong defense against data corruption and inconsistency, which is paramount for data integrity. Defenders should verify that these consistency mechanisms are correctly configured and monitored.
• Hybrid Cloud Data Management: For hybrid cloud strategies involving AI training, CubeFS's approach of caching hot data in public clouds while maintaining full datasets in private clouds helps manage data transfer costs and performance. Defenders need to ensure proper data synchronization and access controls are in place between private and public cloud instances to maintain security and compliance.
• Kubernetes Integration and Management: The CubeFS CSI plugin simplifies deployment and management within Kubernetes environments. Defenders should leverage this integration for consistent resource provisioning, scaling, and monitoring, aligning storage operations with existing cloud-native practices.
• Stay Updated with Community Developments: As an active CNCF project, CubeFS continuously evolves. Defenders should monitor new releases (e.g., upcoming 3.1, 3.5.2 versions focusing on distributed cache, hybrid cloud features) to leverage performance improvements, stability enhancements, and new functionalities that further strengthen their storage posture.

Key Takeaways

• CubeFS is a graduated CNCF project providing a cloud-native, distributed file system with multi-protocol support (S3, HDFS, POSIX), featuring strong consistency based on Raft and Quorum.
• It offers specific solutions for AI storage challenges across data processing, training, and inference, with a focus on high throughput, low latency, and massive scalability for growing model parameters and data volumes.
• The distributed cache system is crucial for efficient model distribution, enabling prompt synchronization, consistency control, and pre-warming capabilities for hybrid cloud AI training and performance optimization.
• Intelligent data tiering significantly reduces storage costs by automatically migrating cold data (e.g., 80% of AI data not read in 3 months) from expensive high-performance media to lower-cost storage, while retaining hot data on faster tiers.
• CubeFS enables robust compute-storage separation, exemplified by its integration with ClickHouse, evolving towards a shared storage model for enhanced reliability, self-repair capabilities, and simplified operational maintenance.
• For performance-critical applications, CubeFS offers an SDK-based client that bypasses FUSE overhead, delivering significantly higher throughput and lower latency, essential for systems like key-value stores with strict P99 latency requirements.

About the Speaker(s)

Leon Chang (Li Chong) is a maintainer of the open-source project CubeFS. He is also in charge of the file system at OPPO, a technology company. This presentation marked his first English speech at a conference.