Kubernetes Meets Climate Science: Building Large-scale Feature De... Armagan Karatosun & Roope Tervo

Kubernetes Meets Climate Science: Building Large-scale Feature Detection Platforms

Speakers: Armagan Karatosun, Online Cloud Data Access Services Expert, EUMETSAT; Roope Tervo

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=pvTRjsSXMi0

Overview

This talk, presented by Armagan Karatosun of EUMETSAT, with contributions from Roope Tervo and a team of climate scientists, delves into the ambitious endeavor of building a large-scale, collaborative platform for Earth system feature detection. The core challenge addressed is the immense volume and diversity of climate data, spanning petabytes across multiple European space agencies, which makes traditional data processing methods impractical. The solution presented leverages Kubernetes and a suite of open-source tools to bring compute resources closer to the data, enabling climate scientists to efficiently identify, label, and analyze critical environmental phenomena.

The significance of this work lies in its direct impact on climate science and operational services. By facilitating systematic feature identification—from tracking tropical storms to monitoring sea ice extent and detecting aviation fog—the platform supports both early warning systems and long-term climate analysis. It transforms how scientists interact with vast datasets, moving from cumbersome data transfers to a streamlined, collaborative environment that integrates machine learning models, annotation tools, and scalable infrastructure. This initiative not only democratizes access to complex climate data but also accelerates the development of advanced applications and machine learning algorithms crucial for understanding and responding to Earth's changing climate.

Background

▶ Watch: Introduction to European Space Agencies and data archives (0:00)

The European public space sector is a collaborative network of institutions, including the European Space Agency (ESA), Europe's gateway to space, and the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT), which operates satellites for monitoring weather, climate, and the environment. EUMETSAT alone manages over 10 petabytes of archive data across more than 300 collections. Complementing these is the European Centre for Medium-Range Weather Forecasts (ECMWF), providing 24/7 operational weather prediction services and hosting a high-performance computing (HPC) facility with a vast meteorological archival and retrieval system (MARS).

The sheer scale and complexity of this data pose significant challenges. Climate data records (CDRs) are long-term, calibrated datasets crucial for climate scientists, but their massive size makes traditional data movement and local processing infeasible. Data sources are numerous, with varied, often binary, formats, necessitating extensive pre-processing and specialized libraries. To overcome these hurdles, the concept of the European Weather Cloud was born: a community cloud designed to move users and compute resources closer to the data, rather than moving the data itself. This infrastructure-as-a-service (IaaS) model provides allocated tenancy for member states and R&D institutions, establishing a common platform for collaboration.

CDRs are vital for understanding long-term changes in Earth's atmosphere, land, cryosphere (ice sheets), and oceans. They provide critical insights, such as analyzing past sea ice extent for climatologists or offering early warnings for phenomena like fog, crucial for aviation safety. High-quality, systematic feature identification is paramount for developing advanced applications and machine learning (ML) algorithms to detect these anomalies. The project outlined in the talk aimed to achieve two main goals: supporting member states in feature detection for early warnings and collecting a global database of identified features over long time series (e.g., 40 years of ice sheet data). The ultimate objective is to establish a community platform for labeling, maintaining, exploring, visualizing, modifying, and exporting these features.

Initial explorations involved evaluating ML models for tropical storm identification. Using Faster R-CNN on over 2,000 images from Japanese meteorological satellites (as Japan experiences more tropical storms than Europe) containing roughly 5,000 storms, and validated against the International Best Track Archive for Climate Stewardship (IBTrACS) database as ground truth, the model showed promise. Storms were categorized using the Dvorak Current Intensity (DCI). While Faster R-CNN offered the best performance among tested models, a significant challenge emerged: the high number of false positives. Many model-detected "tropical storms" were not actual storms according to the IBTrACS database, highlighting the critical need for manual labeling by domain experts. This underscored the complexity of automating climate feature identification and the necessity of human oversight.

Separately, the team explored the Segment Anything Model (SAM) for satellite imagery segmentation. Despite SAM not being specifically trained on satellite data (using the EuroSAT dataset for evaluation), it demonstrated remarkable accuracy in segmenting images. This led to the idea of integrating SAM into an interactive application where users could pinpoint areas of interest and segment them, exporting the annotations to tools like QGIS. This success, combined with the observed need for expert validation in ML-driven feature detection, provided the impetus for developing a comprehensive, collaborative platform that integrates ML models, annotation tools, and scalable infrastructure to empower earth observation scientists.

Key Findings

▶ Watch: European Weather Cloud: Compute near data strategy (3:15)

The development of the Earth system feature detection platform yielded several crucial findings regarding the application of modern IT infrastructure and machine learning to climate science:

1. Effective ML Models for Feature Detection, with Caveats: Faster R-CNN proved to be an effective model for identifying tropical storms, outperforming other tested architectures. Similarly, the Segment Anything Model (SAM) demonstrated impressive capabilities for general satellite imagery segmentation, even when applied to datasets it wasn't specifically trained on (e.g., EuroSAT). These findings validate the potential of transfer learning and pre-trained models in climate science applications.

1. Indispensable Role of Domain Expert Labeling: Despite the advancements in ML models, the problem of false positives and the inherent complexity of climate phenomena necessitate manual labeling and validation by domain experts. The sheer volume of detected features (e.g., thousands of tropical storm candidates) means that fully automated labeling is not yet feasible or reliable, making collaborative annotation platforms a critical component of the ML pipeline.

1. Community Cloud as a Solution to Data Gravity: The European Weather Cloud model, a community-specific IaaS based on OpenStack, successfully addresses the challenge of data gravity. By bringing compute resources into close proximity with petabytes of climate data, it eliminates the need for scientists to download or move massive datasets, significantly accelerating research and development. This model fosters collaboration among member states and institutions.

1. Kubernetes as the Foundation for a Flexible, ML-Friendly Platform: Kubernetes serves as the central orchestration layer, enabling the deployment and management of diverse tools and applications at scale. Its Kubernetes-first approach, prioritizing official operators and Helm charts, allows for a robust, automated, and maintainable platform. This abstraction hides infrastructure complexity from earth observation scientists, letting them focus on their research.

1. Strategic Resource Management for Scarce GPUs: Given the limited availability of specialized hardware (e.g., older A6000 GPUs), the implementation of time slicing for vGPU sharing proved to be a practical and effective strategy. While not as advanced as MPS or MIG, time slicing is suitable for the repetitive, independent data processing jobs common in segmentation tasks, offering a good trade-off in terms of resource utilization and memory isolation.

1. GitOps and Infrastructure as Code Ensure Consistency and Automation: Adopting GitOps principles with Argo CD for managing deployments (JupyterHub, Label Studio, Nucleio, Cloud Native PG) ensures consistency, automates Day 2 operations, and provides a single source of truth for the platform's configuration. This approach is crucial for maintaining a complex multi-application environment with a small ML Ops team.

Technical Deep Dive

▶ Watch: Defining the European Weather Cloud as a community cloud (3:50)

The Earth system feature detection platform is engineered as a flexible, open, and machine learning-friendly environment, built upon a robust Kubernetes foundation within the European Weather Cloud. The overall blueprint involves a structured flow from raw data to visualized features.

Platform Blueprint and Data Flow:

The platform begins with input label data, such as the IBTrACS database, serving as ground truth, and source data from various Earth observation satellites. A critical data preparation step is necessary because source data often arrives in diverse, binary formats. Libraries like satpy are used to process this raw data and generate images, which are then stored in an S3-compatible object storage bucket. The IBTrACS labels are parsed and ingested. For annotation, a community tool called Label Studio is employed, chosen for its extensible API and UI. Within Label Studio, the Segment Anything Model (SAM) assists scientists in accurately labeling detected features. Once annotated, these features are exported to a PostgreSQL database, specifically utilizing the PostGIS extension for geospatial data storage. Finally, these stored features can be visualized through any standard web map service.

Infrastructure as a Service (IaaS):

The underlying IaaS is the European Weather Cloud, which is based on OpenStack. This setup serves as a "pathfinder" project to evaluate the benefits and feasibility for future production environments. The current specifications, while modest (CPU, RAM, block storage), are sufficient for this pilot phase. A key limitation highlighted is the scarcity of GPUs: only two physical Nvidia A6000 GPUs (five years old) are available, presented as vGPU flavors.

Kubernetes Orchestration:

The Kubernetes clusters are managed and provisioned using Rancher. For persistent storage, the OpenStack Cloud Control Manager CSI (Container Storage Interface) plugin is used for read-write-once access, while Longhorn provides read-write-many capabilities. To expose the physical GPUs to Kubernetes pods, the Nvidia GPU Operator is utilized. This operator simplifies installation, maintenance, and bundles essential Nvidia tools, centralizing configuration. Instance groups are configured for autoscaling worker nodes, though the vGPU instance group remains tightly constrained due to resource limitations.

GPU Resource Management and Time Slicing:

Given the extreme scarcity of GPU resources, efficient sharing is paramount. The platform implements time slicing for vGPU sharing. This choice is dictated by hardware limitations; the older A6000 cards do not support more advanced techniques like Multi-Instance GPU (MIG) or Multi-Process Service (MPS). Karatosun justifies time slicing by noting that GPUs are primarily used with SAM for segmenting images, where users run repetitive, mostly unrelated data processing jobs rather than parallel, MPI-style computations. In this scenario, time slicing offers a good trade-off, providing memory isolation and allowing jobs to be reprocessed if data is lost, making MPS an overkill. Resource quotas and GPU utilization monitoring are in place to manage usage effectively.

Design Principles and Tooling:

The platform adheres to strict design principles:

1. Open Source and Open License Ecosystem: Prioritizing open-source tools.
2. GitOps and Infrastructure as Code (IaC): Automating everything with Argo CD and GitOps best practices.
3. Kubernetes-First Approach: Utilizing official Kubernetes operators as the first choice, falling back to well-supported Helm charts if operators are immature or non-existent, and only resorting to custom solutions when no other option is available.

This approach led to the selection of several key open-source components:

• Data Preparation: satpy for converting raw satellite data into image formats, stored in S3-compatible buckets.
• User Interface and Compute: A hosted JupyterHub environment is provided, as scientists are already familiar with it. This hub comes pre-configured with plugins (e.g., Jupyter Git, JupyterLab Git) and necessary libraries, offering GPU processing capabilities to simplify onboarding. Scientists can also bring their own tools if preferred.
• Labeling and Annotation: Label Studio was chosen for its extensibility, API, and customizable UI, making it highly suitable for collaborative image labeling.
• Machine Learning Backend: A custom backend for Label Studio was developed, leveraging Nucleio.
• Function-as-a-Service (FaaS): Nucleio serves as the FaaS framework. It treats code as a whole, building container images at runtime, and executing them as functions (or "processors"). This model aligns well with earth observation science, which often involves sequences of concurrent, independent processors. Nucleio is highly Python-friendly and integrates with other orchestrators like Kubeflow Pipelines.
• Nucleio Challenges: Karatosun acknowledges some downsides: it lacks built-in workload pipelines (requiring users to integrate orchestration logic in JupyterHub or their code), function versioning is not a full model registry, debugging deployed functions can be tricky, and images tend to be large (especially with Python libraries, models, and checkpoints). A current mitigation strategy involves building images outside Nucleio and pushing them to a central registry to avoid runtime builds.
• Database: Cloud Native PG manages the PostgreSQL database with the PostGIS extension, providing robust storage for geospatial features.

GitOps with Argo CD:

The entire platform, encompassing JupyterHub, Label Studio, Nucleio functions, and Cloud Native PG, is managed through GitOps using Argo CD. This ensures consistent deployment, configuration, and Day 2 operations (like database backups) by treating the Git repository as the single source of truth. This automation is critical for a small ML Ops team managing a complex, multi-application environment, providing consistency for end-users.

The Label Studio machine learning backend itself presented an integration challenge, as it typically expects a Docker Compose setup with components like a web application server, SupervisorD, and RQ for job scheduling. The custom solution replaced these with Nucleio-based functions, leveraging Label Studio's ML SDK to build models and present predictions as JSON, which Label Studio readily accepts. This flexibility in output format was a significant advantage, allowing for greater customization and tool interchangeability.

Demo / Proof of Concept

▶ Watch: Evaluating ML models for tropical storm identification (6:00)

The talk included a compelling demonstration of the platform's capabilities, particularly focusing on the integration of Label Studio with SAM for feature annotation.

The demo began by showcasing a Label Studio instance loaded with approximately 1,800 satellite images, pre-uploaded from an S3-compatible bucket. These images were already enriched with "Dorok" labels, derived from the IBTrACS database, serving as the ground truth for different categories of tropical storms. Additionally, the system had integrated "SAM labels," indicating the availability of the machine learning model for auto-annotation.

The speaker navigated to the model integration section, confirming that the SAM model was seamlessly integrated with the Label Studio system. While the UI was demonstrated, it was noted that all these operations could also be performed via an API, highlighting the platform's flexibility for programmatic interaction.

The core of the demo involved the auto-annotation process. The user selected an image, and with "auto annotation" enabled, the system was ready to detect features. By placing a single keypoint on a visible tropical storm, SAM immediately performed the segmentation and generated a precise label, outlining the storm's boundaries. The resulting segmentation was visually impressive, showing strong compatibility with the existing IBTrACS data. The user then accepted this newly generated label, which made it available for extraction and storage in the Earth System Feature Database.

The final part of the demonstration illustrated the power of the integrated platform for visualization. Using data extracted from the database, the talk showed a visualization of Atmospheric Motion Vectors (AMVs). This visualization displayed the movement of individual cloud or water vapor patterns across successive satellite images. The example used was a system that moved slightly over time, and by exporting all the labeled information into the PostGIS database and then visualizing it, the platform could render the "shades" or tracks of the moving cloud or water vapor, effectively visualizing the dynamic nature of atmospheric phenomena. This showcased the end-to-end capability, from raw data and ML-assisted labeling to long-term storage and advanced visualization, empowering climate scientists with critical insights.

Defensive Implications

▶ Watch: Demonstrating Faster R-CNN model performance and detections (7:30)

While the talk "Kubernetes Meets Climate Science" is primarily focused on building a scalable and collaborative platform for scientific research rather than directly addressing security vulnerabilities or defensive strategies, its design principles and architectural choices inherently contribute to a more robust and resilient system.

The "community cloud" model, where resources are allocated to member states and R&D institutions rather than being open to the public, implies a controlled access environment. This is a fundamental security control, limiting who can access sensitive climate data and computational resources, thereby reducing the attack surface compared to a public cloud. The emphasis on GitOps with Argo CD and Infrastructure as Code also brings significant defensive advantages. By maintaining a single, version-controlled source of truth for all deployments and configurations, the platform enhances auditability, ensures consistency, and simplifies recovery from misconfigurations or security incidents. Any unauthorized changes would be immediately visible in the Git repository, allowing for rapid detection and remediation.

Furthermore, the adoption of a Kubernetes-first approach and the use of official operators for managing components like the Nvidia GPU Operator, Cloud Native PG, and Longhorn, mean that the platform benefits from the security best practices and ongoing maintenance of these well-vetted open-source projects. While not explicitly detailed, standard Kubernetes security practices, such as network policies, resource quotas (which are mentioned in the context of GPU utilization), role-based access control (RBAC), and container image scanning, would naturally be applied within such an environment to protect against common threats. The focus on data integrity, through structured labeling and storage in a PostGIS database, is also crucial for scientific data, ensuring that the information used for climate analysis and early warnings is trustworthy and has clear provenance.

Key Takeaways

• Addressing Data Gravity with Community Cloud: The European Weather Cloud effectively solves the challenge of massive climate data by bringing compute resources directly to the data, enabling efficient processing and collaborative research for Earth observation scientists.
• Kubernetes as the Core Orchestration Layer: Kubernetes provides a flexible, scalable, and ML-friendly foundation for deploying and managing diverse open-source tools, abstracting infrastructure complexity for domain experts.
• ML-Assisted Annotation is Crucial, but Human Oversight is Key: Models like Faster R-CNN and SAM significantly accelerate feature detection and segmentation, but the complexity of climate data necessitates manual validation and labeling by domain experts to ensure accuracy and mitigate false positives.
• GitOps for Platform Consistency and Automation: Adopting GitOps with Argo CD ensures a single source of truth for platform configuration, automates deployments and Day 2 operations, and maintains consistency across the multi-application environment.
• Strategic GPU Resource Management: Faced with scarce GPU resources, implementing time slicing for vGPU sharing is a practical and effective solution for scientific workloads characterized by repetitive, independent data processing jobs.
• Empowering Scientists with Integrated Open-Source Tooling: The platform leverages a suite of well-integrated open-source tools (JupyterHub, Label Studio, Nucleio, PostGIS) to provide a comprehensive environment that supports data preparation, ML model application, collaborative annotation, and advanced visualization.

About the Speaker(s)

Armagan Karatosun is an Online Cloud Data Access Services Expert at EUMETSAT, a role he has held since July 2024. Prior to this, he served as a Cloud Computing Engineer at EUMETSAT from 2019 to 2022, and subsequently as a Cloud Computing Engineer at the European Centre for Medium-Range Weather Forecasts (ECMWF) from 2022 to 2024. His extensive background in cloud computing and data access services has been instrumental in the development of the Earth system feature detection platform.

Roope Tervo is a colleague who collaborated with Armagan Karatosun and the team of climate scientists and interns on this project. While he was unable to join the presentation, his contributions were integral to the work discussed. The transcript mentions "my colleague group at Harvard," suggesting an affiliation with the institution.