Kubernetes and AI To Protect Our Forests: A Cloud Native Infrastructure for Wildf... Andrea Giardini

Kubernetes and AI To Protect Our Forests: A Cloud Native Infrastructure for Wildfire Prevention

Speakers: Andrea Giardini, SRE, Overstory

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=1rtyQaTfbdM

Overview

Andrea Giardini, an SRE at Overstory and a Sens ambassador, presented a compelling talk at KubeCon EU detailing how cloud-native technologies, specifically Kubernetes and artificial intelligence, are being leveraged to combat the devastating threat of wildfires. The session, titled "Kubernetes and AI To Protect Our Forests: A Cloud Native Infrastructure for Wildfire Prevention," illuminated the critical role of advanced data processing and workflow orchestration in preemptive wildfire management.

This talk is particularly significant because it addresses a global crisis with increasingly severe consequences for human lives, communities, and natural ecosystems. Wildfires, often unpredictable and sudden, have grown in frequency and intensity, with events like the 2018 Campfire in California burning over 153,000 acres and causing immense destruction. Overstory's innovative approach shifts the paradigm from reactive firefighting to proactive prevention by identifying high-risk areas before fires ignite, thereby offering a scalable and efficient solution to a complex environmental and societal challenge.

The core of Overstory's strategy involves analyzing vast quantities of high-resolution satellite imagery combined with power infrastructure data using machine learning. This data-intensive process demands a robust, flexible, and highly scalable cloud-native infrastructure. Giardini's presentation provides a deep dive into the evolution of Overstory's platform, from its early days relying on Jupyter Hub to its current sophisticated architecture built around Kubernetes and the data orchestration framework Dagster, ultimately demonstrating how cutting-edge technology can be applied to real-world problems with profound impact.

Background

▶ Watch: Introduction and the devastating impact of wildfires (0:00)

Wildfires represent a catastrophic force, with devastating impacts on human lives, communities, and the environment. Giardini highlighted recent statistics, such as a California wildfire event causing 29 deaths, necessitating the evacuation of over 200,000 people, destroying 18,000 homes, and burning over 57,000 acres of land – an area equivalent to a significant portion of London. The 2018 Campfire in California, for instance, consumed an even more staggering 153,000 acres. A surprising and critical statistic revealed is that over 60% of the most destructive wildfires in California are caused by defective power lines. This often-overlooked cause underscores the immense challenge faced by utility companies in maintaining vast and complex electrical infrastructures.

The inspection and maintenance of power line infrastructure is fraught with difficulties. The sheer scale of these networks, which must operate 365 days a year with near 100% uptime, makes traditional visual inspections impractical and costly. Power lines traverse diverse terrains, including private properties, requiring time-consuming permissions and physical access. Furthermore, these infrastructures are constantly exposed to atmospheric agents like storms, thunder, and encroaching vegetation, leading to sudden and catastrophic failures. Giardini cited an example of a small power substation failure near an airport, which disrupted over 1,000 flights and affected 300,000 passengers, illustrating the cascading impact of even minor infrastructure failures.

Recognizing these challenges, Overstory sought a different approach: leveraging very high-resolution satellite imagery and machine learning to derive insights from the sky. The process involves acquiring vegetation data from satellite and aerial images, combining it with energy provider data (pole coordinates, line locations, terrain information), and then generating a risk map. This map enables utility companies to identify where vegetation is too close to power lines, allowing them to proactively trim foliage and prevent wildfires before they start.

Overstory, starting as a small startup, initially adopted a simple yet flexible stack. Kubernetes was chosen for its balance of flexibility and stability, capable of handling diverse workloads ranging from 1 CPU and 4GB of memory to 72 CPUs and half a terabyte of memory. For data processing, Jupyter Hub was the primary tool, allowing data scientists to spin up notebooks with custom resources (larger GPUs, more memory/CPU) beyond their local machines. While effective for initial experimentation, this setup quickly became a "nightmare." Reproducibility and tracking were extremely difficult, leading to frequent issues like "Do you remember which version of Pandas we used a year ago? The new one breaks everything." Workflows were manual, with a one-to-one binding between a client request and a data scientist assigned to manually run and fix Jupyter notebooks.

This highlighted a critical need for automation and improved workflow management. Overstory's two main objectives became clear: eliminate reliance on Jupyter notebooks for production workflows and automate workflow runs to decouple project capacity from the number of data scientists. Efforts to convert Jupyter notebooks to Python packages using tools like NBDE were unsuccessful, reinforcing the understanding that Jupyter notebooks, being essentially giant JSON files, are difficult to maintain, test, and track effectively with Git. This pushed Overstory towards adopting a dedicated data and workflow orchestrator.

Key Findings

▶ Watch: Challenges in preventing wildfires and infrastructure maintenance (3:00)

Overstory's journey to a robust cloud-native infrastructure for wildfire prevention is marked by several key findings and architectural shifts:

1. Transition to Dagster for Workflow Orchestration: The most significant finding was the successful adoption and extensive use of Dagster as the primary data and workflow orchestrator. Dagster, an open-source project, provided the necessary capabilities to move beyond manual Jupyter-based workflows. It offered fast development, easy local testing, Python module-based operations, and a genuinely cloud-native design philosophy. This allowed Overstory to automate complex data processing pipelines, addressing the issues of reproducibility and scalability inherent in their previous setup.

1. Microservice Architecture with Dagster: Overstory implemented Dagster within a classical microservice architecture running on Kubernetes. Key components include:

• Dagit: The web interface, providing users with an overview of pipeline status.
• Dagster Daemon: Responsible for running pipelines based on schedules or conditions, monitoring execution, and handling retries and notifications.
• Code Locations (gRPC servers): Independent deployments, each exposing specific pipelines, schedules, sensors, and assets. This design ensures high isolation, meaning a failure in one team's code location does not impact others, which is crucial for a growing organization.

1. Cloud-Native Features of Dagster: Giardini highlighted two particularly valuable Dagster features:

• IO Managers: This feature abstracts data persistence logic, allowing the same Python code to save data to different locations based on the environment (e.g., S3 or Google Cloud Storage in the cloud, local files during development). This significantly streamlines development and deployment.
• Assets: Dagster's concept of assets focuses on the relationships between different data entities, rather than just how data is built. An asset can be anything from a BigQuery table, a PostgreSQL database, a Snowflake data store, or a file on GCS. This explicit modeling of data dependencies (e.g., Airbyte ingestion feeding dbt projects, which then inform TensorFlow models) provides clear visibility into data flow and lineage, which is essential for complex data pipelines.

1. Custom Library for Kubernetes Integration (Maple): Overstory developed an internal library called Maple to simplify resource allocation for Dagster operations (ops). Maple uses Python decorators to allow developers to specify Kubernetes-specific requirements directly within their Python code, such as requesting a GPU, high-memory nodes (e.g., 32 CPUs, 390 GB RAM), or ephemeral volumes (e.g., 150 GB for scratch space) for temporary files. This abstraction, while still somewhat Kubernetes-specific, significantly streamlined resource management.

1. Flexible Resource Allocation: The platform allows for highly granular resource management. Pipelines can be configured to run entirely within a single pod/node with fixed resources or across multiple pods/nodes, where each step can have its own specific resource requirements. This flexibility is critical for optimizing costs, especially for expensive resources like GPUs, which can be reserved only for the specific steps that require them, rather than for the entire duration of an 8-hour pipeline.

1. Custom Delivery Platform for External Orchestration: While Dagster serves as the core workflow engine, Overstory found limitations in its native scheduling for their unique, long-running, and highly customized delivery processes. They developed a custom delivery platform on top of Dagster, comprising several microservices (starter, router, trigger, watcher) deployed on Google Cloud Run and communicating via Google Pub/Sub. This platform manages the external logic for triggering pipelines, setting parameters, and orchestrating complex, multi-day data flows, effectively decoupling this custom logic from Dagster's core responsibilities.

1. Enhanced Observability: The custom platform integrates seamlessly with observability tools. All Pub/Sub messages are exported to BigQuery, and OpenTelemetry is used to export data to Google Cloud Metrics. These metrics are then visualized in Grafana, providing a comprehensive, end-to-end view of delivery progress, pipeline health, and areas requiring attention or manual intervention. This aggregation of statistics was a key improvement over Dagster's native web interface, which, while good for individual pipeline status, struggled with aggregate metrics like failure rates over time.

1. Achieving "Zero-Touch" Deliveries: The culmination of these architectural changes was the successful execution of their first "zero-touch" end-to-end delivery in just 30 minutes. This achievement, for a recomputed delivery, validated the platform's ability to significantly reduce manual intervention, improve reliability, and accelerate the delivery of critical wildfire risk maps to utility customers.

Technical Deep Dive

▶ Watch: Overstory's solution: satellite imagery and machine learning (4:10)

Overstory's technical architecture is a testament to applying cloud-native principles to solve complex, data-intensive problems. At its foundation lies Kubernetes, specifically Google Kubernetes Engine (GKE), providing the scalable and resilient infrastructure needed to manage highly variable workloads. These workloads can range from small, single-CPU tasks to massive computations requiring 72 CPUs and half a terabyte of memory, demonstrating Kubernetes's flexibility in resource allocation.

The initial phase saw data scientists leveraging Jupyter Hub running on Kubernetes. This setup allowed them to provision custom compute instances with specialized resources like larger GPUs or increased memory, facilitating rapid experimentation. However, the inherent limitations of Jupyter notebooks for production workflows – particularly around reproducibility, version control (Jupyter notebooks are large JSON files, making Git tracking cumbersome), and manual execution – necessitated a shift.

Overstory transitioned to Dagster as their primary data and workflow orchestrator. Dagster is deployed within a microservice architecture on Kubernetes. The user interface, Dagit, provides a web-based dashboard for monitoring. The Dagster Daemon orchestrates pipeline execution, handles scheduling, and manages retries. Critical to the multi-team environment are Code Locations, which are independent gRPC servers. Each code location encapsulates pipelines, schedules, sensors, and assets for a specific team, ensuring high isolation and allowing teams to deploy and manage their workflows without impacting others.

A standout feature of Dagster, crucial for Overstory's diverse environments, is IO Managers. This mechanism abstracts the underlying storage layer. For example, the same Python code can be configured via IO Managers to save output data to Amazon S3 or Google Cloud Storage when running in the cloud, and to local files during local development. This eliminates the need for conditional logic within the application code, simplifying development and ensuring portability.

Another powerful Dagster concept is Assets. Unlike traditional workflow tools that focus on jobs or steps, Dagster's assets model the actual data entities and their dependencies. An asset can be any data artifact – a BigQuery table, a PostgreSQL entry, a Snowflake data store, or a file on GCS. By explicitly defining these assets and their relationships, Overstory gains a clear, graph-based view of data lineage and how data flows through various transformations (e.g., from Airbyte ingestion, through dbt projects, to TensorFlow models for predictions). This focus on data relationships, rather than just individual processing steps, enhances understanding and maintainability of complex pipelines.

To further customize and optimize resource allocation within Dagster, Overstory developed an internal Python library called Maple. Maple uses decorators to augment standard Python functions, transforming them into Dagster "ops" (pipeline steps) with specific Kubernetes resource requests. For instance, a @maple_op decorator can specify that a particular function requires a GPU, or a node with at least 32 CPUs and 390 GB of memory. Importantly, Maple also integrates with Kubernetes ephemeral volumes, a relatively recent feature. These volumes, specified with an amount (e.g., 150 GB), are dynamically created, attached to a pod for its lifetime, and automatically deleted upon pod termination, providing essential scratch space for temporary files without manual volume management.

This granular control over resources allows for two primary execution patterns:

1. Single Pod/Node Pipeline: The entire pipeline with multiple steps runs within a single pod on one node, allocating a fixed set of resources for the duration.
2. Multiple Pods/Nodes Pipeline: Each step in the pipeline runs in its own pod, potentially on different nodes, with tailored resource allocations. This is particularly beneficial for expensive resources like GPUs. Instead of reserving a GPU for an entire 8-hour pipeline, it can be provisioned only for the specific steps that require it, significantly optimizing cost and resource utilization.

Beyond Dagster, Overstory built a custom delivery platform to handle the external orchestration logic and long-running delivery processes that were difficult to model natively within Dagster's scheduling capabilities. This platform consists of several microservices—starter, router, trigger, and watcher—all deployed on Google Cloud Run for serverless execution. Communication between these microservices is facilitated by Google Pub/Sub, a fully managed messaging queueing system, ensuring decoupled and asynchronous processing. The trigger and watcher services specifically interact with the Dagster API to initiate jobs with precise configurations and monitor their progress.

For comprehensive observability, the platform leverages several tools. All messages flowing through Pub/Sub are exported to BigQuery for historical analysis. OpenTelemetry is used to collect and export metrics to Google Cloud Metrics. These diverse data sources are then aggregated and visualized in Grafana dashboards. This provides Overstory teams with an end-to-end view of the delivery process, highlighting the status of each step, identifying bottlenecks, and flagging any steps requiring manual intervention, thereby offering critical insights that Dagster's native interface couldn't provide at an aggregate level.

Future technical directions include further abstracting Kubernetes details within the Maple API to simplify the developer experience, exploring GKE node pool auto-provisioning and Kubernetes dynamic resource allocation to optimize resource usage and break the one-step-one-node assumption, and continuously improving monitoring and observability capabilities to predict and prevent under- or over-provisioning of resources, a challenge given the unpredictable nature of their data inputs. Andrea also mentioned looking into Dagster's asset checks feature for ensuring data quality and preventing breaking changes in downstream pipelines.

Demo / Proof of Concept

▶ Watch: Initial tech stack: Kubernetes with Jupyter Hub (6:20)

While the talk did not feature a live, interactive demonstration in the traditional sense, Andrea Giardini presented a compelling "proof of concept" through the achievement of a significant milestone for Overstory. The speaker proudly shared a screenshot illustrating their first "zero-touch" end-to-end delivery completed in just 30 minutes.

This achievement was not for a new delivery but a recomputation of a previous one, allowing for direct comparison of outputs to ensure accuracy. This outcome represents the culmination of nearly a year's worth of dedicated work by Overstory's platform team, working closely with the SRE team. It validated the entire cloud-native infrastructure, demonstrating its ability to drastically reduce manual intervention and accelerate the processing of critical data for wildfire prevention.

The Grafana dashboard, showcased in the technical deep dive, serves as a visual testament to the platform's capabilities, providing real-time visibility into the progress of each delivery. This dashboard allows teams to monitor every step of the complex process, identify any steps requiring attention or failing more than usual, and pinpoint where manual intervention might still be needed. The ability to track a delivery end-to-end, from initial image acquisition and infrastructure data ingestion to final risk map generation, provides the necessary transparency and control that the previous manual, Jupyter-based workflows lacked. This rapid, automated, and observable delivery of actionable insights is the ultimate proof of concept for Overstory's advanced cloud-native solution.

Defensive Implications

▶ Watch: Early challenges with reproducibility in Jupyter notebooks (8:00)

The "defensive implications" of Overstory's work are profound, though they pertain to environmental and infrastructure defense rather than cybersecurity. The primary defensive action is proactive wildfire prevention through intelligent vegetation management, a stark contrast to the reactive and often devastating efforts of wildfire containment.

1. Shift from Reactive to Proactive Prevention: By leveraging high-resolution satellite imagery and machine learning, Overstory enables utility companies to identify areas of elevated wildfire risk before an incident occurs. This allows for targeted vegetation trimming and maintenance around power lines, directly addressing the root cause of over 60% of destructive wildfires in California. This proactive approach saves lives, protects homes, businesses, and critical infrastructure, and preserves natural ecosystems.

1. Enhanced Infrastructure Resilience: The platform helps utility companies maintain the integrity of their vast and complex power grid infrastructure. By identifying encroaching vegetation that could lead to short circuits or equipment damage, Overstory's solution contributes to the resilience and uptime of the electrical network, minimizing the risk of catastrophic power outages and the subsequent economic and social disruptions.

1. Cost and Resource Optimization: Traditional visual inspections of power lines are extremely costly and time-consuming, often requiring access to private property. Overstory's satellite-based analysis offers a significantly more efficient alternative, reducing operational expenses for utility companies. This optimization of resources means that maintenance budgets can be more effectively allocated to actual intervention, rather than just costly identification.

1. Reduced Human Error and Increased Reliability: The automation provided by Dagster and Overstory's custom delivery platform minimizes human intervention in the data processing pipeline. This significantly reduces the risk of human error, ensuring that risk maps are generated consistently and reliably. The "zero-touch" delivery capability means that critical insights are delivered faster and with greater accuracy, leading to more dependable prevention efforts.

1. Scalability of Prevention Efforts: The cloud-native architecture, built on Kubernetes and Google Cloud services, provides the necessary scalability to analyze vast geographical areas and process enormous datasets. This means that wildfire prevention efforts are no longer limited by manual capacity but can scale to meet the demands of entire regions or even countries, offering a truly comprehensive defensive strategy against wildfires.

In essence, Overstory's platform provides utility companies with an intelligent, data-driven defense mechanism. It empowers them to anticipate threats, act decisively, and ultimately protect communities and environments from the devastating impact of wildfires, transforming a reactive crisis response into a systematic and preventative strategy.

Key Takeaways

• Wildfires pose a significant and growing threat, with defective power lines being a major cause, particularly in regions like California. Traditional inspection methods are costly, time-consuming, and insufficient for the vast scale of infrastructure.
• Overstory leverages high-resolution satellite imagery (up to 15cm resolution) combined with machine learning to proactively identify areas where vegetation poses a wildfire risk to power lines, enabling targeted preventative action.
• The transition from manual, Jupyter Notebook-based workflows to an automated, cloud-native infrastructure built on Kubernetes and Dagster was critical for improving reproducibility, scalability, and efficiency in processing vast datasets.
• Dagster's cloud-native features, such as IO Managers (abstracting data storage) and Assets (modeling data dependencies), are fundamental to managing complex data pipelines, ensuring data lineage, and simplifying development.
• Overstory developed a custom delivery platform using Google Cloud Run and Pub/Sub to orchestrate long-running, complex pipeline triggers and configurations externally from Dagster, providing enhanced control and end-to-end observability via BigQuery, OpenTelemetry, and Grafana.
• The integrated system has achieved "zero-touch" end-to-end deliveries in as little as 30 minutes, drastically reducing processing time and manual intervention, leading to faster, more reliable, and scalable wildfire prevention efforts.

About the Speaker(s)

Andrea Giardini is an accomplished Site Reliability Engineer (SRE) at Overstory, the company at the forefront of leveraging AI and cloud-native solutions for wildfire prevention. Beyond his professional role, Andrea is a Sens ambassador, highlighting his commitment to promoting secure and reliable systems. He is also actively involved in the community, frequently organizing community events and conferences across Europe. His expertise spans cloud infrastructure, SRE practices, and the application of advanced technologies to solve real-world problems.