Image Snapshotters for Efficient Container Execution in Particle P... Clemens Lange & Valentin Volkl

Image Snapshotters for Efficient Container Execution in Particle Physics

Speakers: Clemens Lange, Physicist, Paul Scherrer Institute (PSI); Valentin Volkl, Physicist turned Software Developer, CERN

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=Dc6S4vU9GiM

Overview

This talk, presented by Clemens Lange and Valentin Volkl, delves into the critical role of image snapshotters in optimizing containerized workloads, with a particular focus on the demanding environment of particle physics research at institutions like CERN and the Paul Scherrer Institute (PSI). The core problem addressed is the inefficiency inherent in traditional container image distribution, where a significant portion of downloaded data often goes unused. The speakers introduce and compare various "lazy pulling" snapshotters that integrate with containerd, demonstrating how these technologies enable near-instantaneous container startup and drastically reduce network bandwidth and storage requirements.

The talk is highly relevant not only to the scientific computing community, which grapples with immense datasets and long-running experiments requiring stable software environments over decades, but also to any organization deploying containers at scale. By showcasing practical implementations and benchmark results, Lange and Volkl highlight the operational advantages of lazy pulling, such as improved resource utilization, faster deployment cycles, and enhanced developer experience. Their work underscores a fundamental shift in how container images can be managed and executed, moving beyond the limitations of full image downloads to a more intelligent, on-demand approach.

The speakers, both physicists by background, bring a unique perspective, grounding the technical discussion in the real-world challenges of high-energy physics. Clemens Lange is a particle physics researcher at PSI, working on the CMS experiment at the Large Hadron Collider (LHC). Valentin Volkl is a physicist turned software developer at CERN, where he developed the CERNVM File System (CVMFS), a technology central to one of the discussed snapshotter implementations. Their combined expertise provides a comprehensive view of both the scientific necessity and the engineering solutions.

Background

▶ Watch: Introduction to particle physics challenges at CERN/PSI (0:00)

Particle physics experiments, such as those conducted at CERN's Large Hadron Collider (LHC) with detectors like CMS (Compact Muon Solenoid) and at PSI's Ring Cyclotron, represent some of the most ambitious and long-running scientific endeavors globally. The CMS experiment, for instance, is a massive detector—15 meters high and 25 meters long—that captures up to 40 million particle collision events per second, operating almost 24/7. These experiments generate unique and invaluable data, leading to discoveries like the Higgs boson, and are planned to continue data acquisition until at least 2041, spanning several decades of operation.

This extended operational lifespan presents a significant challenge for software and infrastructure management. Experiments often begin with one operating system version (e.g., Red Hat Enterprise Linux 5 in 2009) and must continue to run their analysis software on systems that evolve through multiple generations (e.g., RHEL 6, 7, and now RHEL 9). Maintaining compatibility and security for such long-running, critical workloads across shifting underlying platforms is extremely difficult. The only viable solution, as the speakers emphasize, is containerization. By packaging applications and their dependencies into isolated containers, scientists can ensure that their analysis pipelines remain stable and executable regardless of the host system's Linux distribution or patch level, thereby insulating their work from the continuous evolution of IT infrastructure.

However, containerization introduces its own set of inefficiencies, especially when dealing with the colossal software frameworks common in scientific research. On average, when a container is run, only about 15% of the actual data downloaded from the registry is used; the remaining 85% is effectively wasted bandwidth and storage. While aggressive container minimization can help, it's often not feasible for complex scientific applications that require a broad range of functionalities. This problem is exacerbated in distributed computing environments like the Worldwide LHC Computing Grid (WLCG), where container images are pulled across numerous worker nodes globally. The need for efficient execution of containerized workloads, particularly in terms of data usage and startup time, led to the exploration of lazy pulling mechanisms implemented through containerd snapshotters.

Lazy pulling is a technique where container layers are not fully downloaded upfront. Instead, only the metadata (like an index) is fetched initially, and the actual file data is downloaded on-demand as it is accessed during container execution. This approach drastically reduces the initial data transfer and startup time, as the container can begin running almost immediately, with subsequent performance dependent on how quickly required data can be streamed.

Key Findings

▶ Watch: Highlighting the 85% unused data in typical containers (4:40)

The central finding of the talk is that lazy pulling snapshotters offer a transformative solution to the inefficiencies of traditional container execution, especially for large, complex images. The speakers benchmarked several such snapshotters against the conventional OverlayFS snapshotter, demonstrating clear advantages across key metrics.

First, lazy pulling snapshotters enable near-instantaneous container startup. While OverlayFS might take 15 seconds or more to pull and prepare a moderately sized image due to full downloads, lazy pulling snapshotters make the container available for execution almost instantly. This significantly improves developer experience for interactive tasks and reduces latency for batch jobs.

Second, these snapshotters achieve a drastic reduction in downloaded data. Benchmarks showed that lazy pulling snapshotters downloaded 10% or even less of the data compared to OverlayFS for the same Python 3.9 image. Even for a more complex ROOT image with varying workloads, significant data reduction was observed. This translates directly into lower bandwidth costs, reduced storage requirements on worker nodes, and faster overall data transfer times, especially beneficial in distributed computing environments.

Third, a crucial benefit highlighted is that lazy pulling snapshotters obviate the need for aggressive image optimization by users. With traditional methods, users are often encouraged to meticulously craft minimal Dockerfiles to reduce image size. However, for large scientific frameworks or general-purpose images, such optimization is often impractical or detrimental to flexibility. Lazy pulling handles this implicitly by only fetching what's needed, freeing users from this burden.

The talk also identified several prominent lazy pulling snapshotter implementations: Starg, Sochi (Seekable OCI), and the CERNVM File System (CVMFS) snapshot. While each has distinct technical underpinnings and trade-offs, they all deliver comparable performance benefits in terms of data reduction and startup speed. A critical "no-risk" finding is that all lazy snapshotters are designed with fallback mechanisms; if an image or registry does not support lazy pulling, they seamlessly revert to traditional full image downloads, ensuring operational robustness. The choice among them depends on specific infrastructure, registry support, and image management workflows.

Technical Deep Dive

▶ Watch: Lazy pulling snapshotters: efficient container execution solution (6:00)

The core of efficient container execution with lazy pulling lies in how different snapshotters manage and access image layers. Containered's plugin system allows for various implementations, each with unique approaches to on-demand data retrieval.

Starg Snapshot

The Starg snapshot (from "searchable tar") is one of the pioneering lazy pulling solutions. Its fundamental idea is to transform standard container image layers into a searchable tar format. This requires that images pushed to the registry are rebuilt or converted. Starg uses an extension to the standard gzip format, allowing individual files within a layer blob to be extracted without decompressing the entire layer. Traditionally, to get a single file from a tar.gz layer, the entire layer must be extracted. Starg, by packaging layers into smaller, addressable units or providing an internal index, enables the extraction of specific files individually.

A key implication of Starg is the necessity to rebuild images into this extended Starg format. This conversion step can be a significant barrier to adoption, as it requires changes to existing image build pipelines and potentially a re-conversion of entire image repositories. To mitigate performance slowdowns that can occur when fetching many small, disparate files on demand, Starg offers a mechanism for file prioritization, allowing frequently accessed files to be prefetched or prioritized for download, thereby reducing perceived delays during execution.

Sochi Snapshot (Seekable OCI)

The Sochi snapshot emerged as a fork of Starg, specifically designed to address the build-time conversion requirement. Sochi achieves lazy pulling without modifying the actual image layers themselves. Instead, it adds a separate index artifact alongside the image in the container registry. This index contains metadata about the files within each layer, allowing the snapshotter to pinpoint and fetch only the necessary data chunks on demand.

The primary advantage of Sochi is that it does not require a build-time conversion step. The index artifact can be generated and added after the fact to an existing OCI-compliant image. This significantly lowers the barrier to adoption for users with existing image repositories. Sochi also allows for optimization by configuring a minimum layer size for indexing, typically around 10 megabytes. Layers smaller than this threshold might be downloaded entirely rather than indexed, as the overhead of indexing could outweigh the benefits for very small layers.

Unlike Starg, Sochi deliberately avoids file prioritization. This design choice has the benefit of improving layer sharing across different images. If layers are not reordered or modified for prioritization, they can be more easily deduplicated and shared among multiple container images, leading to further storage and bandwidth savings. A notable caveat for Sochi is its reliance on registry support for additional artifacts. While registries like Harum support this feature, others like GitLab's registry do not, which can limit its applicability depending on the chosen registry provider.

CERNVM File System (CVMFS) Snapshot

The CVMFS snapshot takes a different approach, leveraging the existing CERNVM File System (CVMFS), a global, read-only file system specifically engineered for software distribution in scientific computing. CVMFS is an on-demand streaming service for scientific software, akin to a "Spotify for software." It presents a global directory (/cvmfs) on user machines, and files accessed within this directory are transparently downloaded and cached from remote servers.

CVMFS is implemented as a File System in User Space (FUSE), making it easily deployable across diverse distributed computing environments without requiring kernel modifications. It is heavily optimized for software distribution, featuring:

• An object store that backs the file system, enabling deduplication at both file and sub-file chunk levels. This means if multiple container images share common libraries or binaries, CVMFS only stores and transfers unique chunks.
• Multiple levels of caching to minimize network round trips.
• Uses plain HTTP transport for wide compatibility and ease of deployment.
• Robust verification of data integrity to ensure software authenticity.
• Compression of data to further reduce storage and bandwidth.

CVMFS is a mature project, over 15 years old, hosting approximately 4 billion files from numerous scientific experiments (LHC, LSST, ESA, HPC projects like Easy). A significant repository, unpack.ch, specifically stores unpacked container images. The CVMFS snapshot for containerd works by providing tooling to take a whole container image and unpack it into a CVMFS registry. This requires dedicated infrastructure for the CVMFS servers and the unpacking workflow. Once unpacked, the container's contents are available via CVMFS, allowing containerd to access them on demand. This approach is conceptually similar to the Naida snapshot, which also relies on an external content distribution service. For large files, CVMFS internally chunks them into configurable sizes, which significantly aids in network efficiency and resilience.

Demo / Proof of Concept

▶ Watch: Explaining the mechanics of lazy pulling and on-demand data (6:15)

The talk included a compelling recorded demonstration and rigorous benchmarking to illustrate the practical benefits of lazy pulling snapshotters. The demo utilized a custom application designed to visualize network activity, the overlayfs snapshot directory, and the cvmfs cache during container operations.

First, a standard Docker Hub Python 3.9 image was pulled using the traditional OverlayFS snapshotter. The demonstration clearly showed a download of approximately 400 megabytes of data, which was then fully written into the overlayfs snapshot directory. This process took a noticeable amount of time, reflecting the full image download and unpacking.

Next, the same image was pulled and run using the CVMFS snapshotter. The pull command was virtually instantaneous, as it only needed to fetch the image manifest and verify that the layers were available on the CVMFS backend. Critically, no data was downloaded to the local cvmfs cache or snapshot directory at this stage. When the container was then executed, and a non-trivial operation like an import statement in Python was performed, the demo showed a burst of network activity and data being written into the cvmfs cache. This visually confirmed the on-demand downloading mechanism: files and even chunks of files were fetched only when they were actively accessed by the running container. While there might be a slight latency for interactive use during the initial access of new files, the overall efficiency for long-running workloads is significantly improved.

Beyond the visual demo, quantitative benchmarks were presented for a Python 3.9 image and a more complex ROOT data analysis software image.

For the Python 3.9 image:

• Data Downloaded: OverlayFS downloaded the full compressed image. All lazy pulling snapshotters (Starg, Sochi, CVMFS) downloaded 10% or less of that data. Minor differences between the lazy snapshotters were attributed to their specific implementation details and configurations.
• Time to Execute (trivial workload like python print hello): OverlayFS took around 15 seconds due to network throughput limitations for the full download. The lazy pulling snapshotters were almost instantaneous in becoming ready for execution. While the total execution time for the trivial workload might be slightly longer for lazy pulling due to the on-demand fetching, the crucial metric of "time until ready to execute" was drastically reduced.

For the ROOT image (a widely used CERN data analysis framework) with varying workload complexities:

• Data Downloaded:
• Simply getting a bash shell ready incurred minimal download.
• Starting python downloaded a bit more.
• Importing the root Python bindings (import ROOT) triggered a significant download, consuming around 20% of the overall image data.
• Performing a task like filling a histogram in a random way further increased downloads.
• Again, the lazy pulling snapshotters showed comparable performance in terms of data downloaded and time to readiness, consistently outperforming OverlayFS. The benchmarks highlighted that even for "optimized" images, lazy pulling offers benefits, and that import statements in complex applications can be surprisingly "expensive" in terms of data access.
• Time to Execute: While the overall execution time for long-running workloads might be similar or slightly longer for lazy pulling due to the on-demand nature, the advantage of much faster startup remains. For short-running workloads, lazy pulling still provides a benefit by allowing the workload to start sooner.

Defensive Implications

▶ Watch: Survey of available containerd lazy pulling snapshotters (7:15)

While the talk primarily focuses on performance and efficiency, the adoption of lazy pulling snapshotters has several important defensive implications for container security and operational resilience.

Firstly, the significant reduction in downloaded data directly translates to reduced network bandwidth and storage footprint. This can mitigate certain types of denial-of-service attacks or resource exhaustion scenarios where adversaries might attempt to flood systems with large image pulls. Less data residing locally also means a smaller overall attack surface and potentially less sensitive data exposed if a node were compromised.

Secondly, the ability to achieve near-instantaneous container startup is a boon for operational security. Faster startup times facilitate quicker deployment of patched images, enabling organizations to respond more rapidly to newly discovered vulnerabilities (e.g., zero-day exploits in base images or application dependencies). This agility is crucial in maintaining a strong security posture.

Thirdly, lazy pulling simplifies the management of legacy software environments. As highlighted by the particle physics use case, experiments must run software developed decades ago on modern, secure Linux distributions. Containerization is the key enabler, and lazy pulling ensures that these large, often unoptimized legacy images can be run efficiently without compromising the security of the underlying host systems. This allows critical scientific (or business) operations to continue securely, even when their software stack is inherently complex and difficult to refactor for modern minimal images.

Finally, the fallback mechanism inherent in all lazy snapshotters (reverting to traditional full downloads if lazy pulling fails) ensures operational robustness. This prevents potential security incidents or outages that could arise from misconfigurations or unsupported image formats, providing a layer of resilience. However, organizations should carefully consider registry support for features like additional artifacts (for Sochi) or the infrastructure requirements for CVMFS, as these choices can impact the seamless adoption and security of their container ecosystem. By embracing lazy pulling, organizations can achieve a more secure, efficient, and responsive container deployment strategy.

Key Takeaways

• Containerd's plug-in system is a powerful enabler for extending container runtime functionalities, allowing for innovative solutions like lazy pulling snapshotters.
• Lazy pulling snapshotters drastically improve container efficiency by enabling near-instantaneous startup and significantly reducing the amount of data downloaded (often 85% or more unused data is avoided).
• These technologies eliminate the need for users to aggressively optimize container images, simplifying workflows and making it easier to manage large, complex software frameworks without compromising performance.
• Multiple mature and comparable lazy pulling options exist, including Starg (requiring image conversion), Sochi (using separate index artifacts, registry-dependent), and the CVMFS snapshot (leveraging a dedicated content distribution system).
• There are no significant disadvantages to using lazy pulling snapshotters; they are designed with fallback mechanisms to ensure robustness and consistently offer advantages in efficiency, resource utilization, and user experience.
• Organizations should evaluate their specific infrastructure, registry capabilities, and existing image management workflows to select the most suitable lazy pulling snapshotter for their environment.

About the Speaker(s)

Clemens Lange is a physicist deeply involved in particle physics research. He conducts his research at the Paul Scherrer Institute (PSI), Switzerland's national lab for natural sciences and engineering. His work specifically focuses on one of the experiments at the Large Hadron Collider (LHC), the CMS (Compact Muon Solenoid) experiment, contributing to the analysis of high-energy particle collisions.

Valentin Volkl is also a physicist by training who transitioned into a software development role. He is a key developer at CERN, where he was instrumental in creating the CERNVM File System (CVMFS). This file system is a critical component for software distribution in scientific computing and has been adapted to implement a containerd snapshotter, playing a central role in the efficient container execution discussed in this talk.