In the Eye of Falco: Tr... Luca Guerra, Jason Dellaluce, Aldo Lacuku, Leonardo Grasso, Aurélie Vache

In the Eye of Falco: Tr... Luca Guerra, Jason Dellaluce, Aldo Lacuku, Leonardo Grasso, Aurélie Vache

Speakers: Luca Guerra; Jason Dellaluce; Aldo Lacuku; Leonardo Grasso; Aurélie Vache

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=Cn8xvysLWVg

Overview

Falco, the pioneering cloud-native runtime security tool and a graduated CNCF project, is akin to a security camera for your cloud environments, diligently detecting security threats and delivering timely alerts. This talk, presented by a team of Falco maintainers including Leonardo Grasso, Luca Guerra, Jason Dellaluce, Aldo Lacuku, and Aurélie Vache, provided a comprehensive update on the project's latest advancements and strategic roadmap. It delved into significant improvements across performance, maintainability, and ecosystem expansion, underscoring Falco's commitment to adapting to the evolving landscape of cloud-native security.

The presentation highlighted several key areas of development, including a streamlined container image strategy, enhanced ecosystem plugins for various cloud platforms, a groundbreaking new plugin for container metadata enrichment, the introduction of a Kubernetes-native Falco operator, and a declarative event generator for robust testing. These updates collectively aim to make Falco more efficient, easier to manage, and more powerful in identifying anomalous behaviors within kernel and cloud events. The speakers emphasized how these innovations empower users to tailor Falco to their specific use cases, reduce operational overhead, and bolster their overall supply chain security posture.

The importance of this talk lies in its demonstration of Falco's continuous evolution to meet the complex demands of modern cloud-native infrastructures. By tackling issues like image bloat, vulnerability fatigue, and the need for Kubernetes-native management, Falco is solidifying its role as an indispensable tool for runtime security. The advancements presented offer tangible benefits for DevOps and security engineers, enabling more precise threat detection, automated responses, and a more resilient security framework across diverse cloud and Kubernetes deployments.

Background

▶ Watch: Introduction to Falco: Cloud Native Runtime Security Tool (0:00)

Falco operates by monitoring kernel or cloud events, tapping into the kernel's activity through either an eBPF probe or a kernel module. These events are then fed into a powerful rules engine, which matches them against a set of predefined rules designed to identify anomalous or malicious behaviors. Upon detection, Falco immediately emits an alert, providing critical real-time security insights. Beyond kernel-level monitoring, Falco boasts a robust plugin system that allows it to ingest data from diverse cloud sources, such as Kubernetes audit logs, and integrate with third-party tools like Octo or Githan. The output of Falco alerts is equally flexible, with an ecosystem tool called Falco Sidekick enabling delivery to over 50 destinations, including Slack, Kafka, and various databases.

Historically, Falco faced several challenges that are common in rapidly evolving cloud-native environments. One significant issue was the size and complexity of its container images. The "good old big image" exceeded 1 GB, containing numerous compilers and tools primarily needed for building the kernel module or legacy eBPF probe on the fly. This led to larger deployment footprints and, critically, a high number of reported vulnerabilities by Software Composition Analysis (SCA) tools like Trivy or Grype, many of which were false positives stemming from unused components. This created a considerable burden for DevOps and security engineers responsible for maintaining compliance.

Furthermore, Falco's core component for collecting container metadata, while foundational, was written in C++. This presented maintainability challenges due to the age of the code and the lack of native SDKs for container runtimes like Docker or CRI, necessitating in-house development and increasing the cost to the community. The synchronous nature of metadata lookups also impacted performance and reliability, as Falco couldn't stall its event processing pipeline waiting for container information. This often resulted in the first few events from short-lived containers lacking crucial enrichment, a significant gap in security context. Finally, managing Falco deployments in Kubernetes, particularly across multiple clusters or with varied configurations, was constrained by the limitations of traditional Helm charts, which offered limited flexibility for artifact management and multi-instance deployments. These underlying issues formed the backdrop against which the latest Falco developments were conceived and implemented.

Key Findings

▶ Watch: Agenda: Latest Falco features, ecosystem, and upcoming developments (2:05)

The KubeCon EU talk unveiled a series of significant advancements in Falco, collectively enhancing its performance, usability, and security capabilities across cloud-native environments.

Firstly, a major stride was made in container image streamlining. By adopting minimal Wolfie distro images as the default, Falco dramatically reduced image sizes and, more importantly, slashed the number of reported vulnerabilities by SCA tools from typically 135 to a mere 1. This was facilitated by the new eBPF probe using CO-RE (Compile Once – Run Everywhere), eliminating the need for on-the-fly kernel module compilation. This not only eases compliance burdens but also improves overall security posture.

Secondly, performance received a notable boost. Low-level optimizations, including a switch to Clang and Zig compilers and a change in the memory allocator, resulted in up to a 10% improvement in event throughput. This seemingly modest percentage translates into substantial gains for users processing millions of events per second. New rule operators, such as the in operator, also made rules more expressive.

Thirdly, the Falco ecosystem expanded with new projects and plugins. Falco Actions emerged as a solution for supply chain security, integrating Falco into CI/CD pipelines (specifically GitHub Actions) to detect suspicious events during the build process. Falco Talon, Falco's Kubernetes-native response engine, gained new capabilities, including support for GCP actions and the ability to perform live captures of kernel activity for offline analysis. Crucially, new plugins were introduced for AKS and Vcloud MKS audit logs, extending Falco's reach into more managed Kubernetes environments.

Fourthly, a new container metadata plugin was developed to address long-standing issues with container enrichment. This Go-based plugin leverages native SDKs and connects to container created events, providing instant and reliable metadata enrichment even for short-lived containers. This modular approach significantly improves maintainability, testability, and the accuracy of security alerts.

Fifthly, the Falco Operator was introduced as a Kubernetes-native management solution. This operator provides declarative control over Falco deployments, enabling multi-instance management and flexible artifact management (rules, configs, plugins) from various sources like OCI registries or ConfigMaps, with support for overriding and priorities. This addresses the limitations of Helm-based deployments, offering greater flexibility and easier configuration maintenance.

Finally, the Event Generator received a major upgrade with a declarative test suite. Users can now define complex kernel activity simulations in YAML, including process trees, resources, and syscall steps, along with expected outputs. This transformation makes the Event Generator an invaluable tool for rigorously testing Falco rules and validating the Falco engine itself.

Technical Deep Dive

▶ Watch: Significant performance improvements: up to 10% throughput increase (5:30)

The technical innovations presented in the talk underscore Falco's strategic evolution towards enhanced performance, modularity, and cloud-native integration.

The most foundational technical change lies in Falco's container image strategy. Historically, Falco relied on a monolithic image exceeding 1 GB, necessitated by the requirement to compile the kernel module or legacy eBPF probe at runtime. This "big image" contained compilers and development tools, contributing to a large attack surface and a high number of reported vulnerabilities by SCA tools. The shift to a minimal Wolfie distro as the default image, coupled with the adoption of the new eBPF probe utilizing CO-RE (Compile Once – Run Everywhere), is a game-changer. CO-RE allows eBPF programs to be compiled once and run on different kernel versions, eliminating the need for compilers within the runtime image. This dramatically reduces the image size and, as demonstrated, cuts down false positive vulnerability alerts from 135 to just 1 in typical SCA scans. This not only improves security by reducing the attack surface but also significantly reduces the operational burden on security and DevOps teams. Further low-level optimizations include changing the default compiler to Clang and Zig, and modifying the underlying memory allocator, which collectively yield a 10% improvement in event throughput. Additionally, new rule operators like the in operator have been added, allowing for more expressive and concise rule definitions, mirroring capabilities found in many programming languages.

The expansion of ecosystem plugins for cloud audit logs is another critical development. For AKS clusters, the new plugin requires configuring Azure Event Hubs to receive audit logs and an Azure Blob Storage for checkpointing. The Falco plugin then reads from the Event Hub interface, using the Blob Storage to maintain its position. Installation is simplified via falcoctl artifact install, requiring specific environment variables for Event Hub and Blob Storage. Similarly, for Vcloud MKS, the plugin integrates with the LDP (Log Data Platform). Users must create an LDP, define a data stream, connect the MKS cluster to it, and retrieve a WebSocket URL. The Falco plugin then establishes a WebSocket connection to the LDP, reading audit logs in real-time. Both plugins are designed for easy deployment through falcoctl commands or Helm charts, with detailed configuration examples provided in falco.yaml files. These plugins abstract away the complexities of integrating with different cloud provider logging mechanisms, making it easier for users to extend Falco's runtime security monitoring to a broader array of managed Kubernetes services.

A major architectural improvement is the new container metadata plugin. The previous C++ implementation, integrated directly into Falco's core, suffered from maintainability issues, a lack of native SDKs for container runtimes, and a synchronous lookup model that caused performance bottlenecks due to mutex contention. This often resulted in delayed or missing metadata enrichment, especially for short-lived containers. The new plugin, developed in Go, leverages the rich Go ecosystem for native SDKs to interact with Docker and CRI interfaces. Crucially, it connects to container created events, which occur before the container starts executing its first syscalls, and also monitors container stop events. This allows Falco to know about a container's existence and lifecycle much earlier and more reliably. The modular nature of the plugin system allows for a hybrid approach, where performance-critical paths might still be in C++ while integration logic resides in Go. The outcome is instant data enrichment, improved testability, new Falco fields for rules, and a more linear CPU usage profile. While it introduces minor breaking changes for static builds and exported metrics prefixes (adhering to plugin system standards), the plugin will be bundled by default in Falco packages, ensuring a seamless user experience for most installations.

The introduction of the Falco Operator represents a paradigm shift in managing Falco within Kubernetes. Traditional Helm chart deployments are limited to single instance types (e.g., DaemonSet or Deployment) and struggle with maintaining consistent configurations across multiple clusters, often leading to configuration drift through manual values.yaml updates. The Falco Operator, a Kubernetes controller, watches FalcoInstance Custom Resource Definitions (CRDs). These CRDs define Kubernetes-specific configurations (like pod template specs, deployment types) but not Falco's internal configurations. Instead, the operator injects an Artifact Operator as a sidecar container. This sidecar is responsible for managing Falco's artifacts (rules files, configurations, plugins) using specific CRDs (e.g., RulesFile, Config, Plugin). Artifacts are fetched from various sources—OCI registries for centralized, versioned artifacts; Kubernetes ConfigMaps for cluster-managed configurations; or raw YAML for inline specifications—and made available to Falco via a shared volume mount. The operator supports multi-instance management, allowing different Falco deployments (e.g., one for node-level syscall monitoring, another for cloud audit logs) with distinct configurations to coexist and be managed declaratively. A powerful feature is flexible artifact management with priorities. For example, a user can retrieve a default rule set from an OCI registry, then use an inline YAML configuration in a ConfigMap to override specific macros or conditions within those rules. Priorities ensure that higher-priority sources or inline configurations take precedence, enabling fine-grained control and customization without modifying original artifact sources.

Finally, the Event Generator has evolved into a sophisticated declarative test suite. Previously, test actions were hardcoded in Go. Now, users can define complex kernel activity simulations in YAML. A test definition includes a name, the rule it targets, a context to set up a process lineage (process tree with arguments, capabilities, and names), resources (e.g., client-server setups), and steps that trigger specific syscalls. This generates real kernel events that Falco can detect. Advanced features include binding values from resources to syscall arguments, a matrix for generating multiple events with varied inputs, and the ability to define expected output values that the Event Generator can verify against Falco's actual alerts. This transformation makes the Event Generator an indispensable tool for developing, testing, and validating Falco rules and the Falco engine's detection capabilities in a reproducible and automated manner.

Demo / Proof of Concept

▶ Watch: Falco Actions: CI/CD supply chain security with GitHub (6:25)

While the presentation did not feature a live, interactive demonstration of each new component, the speakers provided clear conceptual walkthroughs and code snippets that illustrate the functionality and ease of use for the new features. For instance, the installation of the AKS and Vcloud MKS plugins was described using the falcoctl artifact install command, accompanied by examples of the required falco.yaml configuration content, including placeholders for event hub and blob storage fields. Similarly, the Falco Operator's interaction with FalcoInstance CRDs and the artifact operator's management of RulesFile and Config CRDs were explained with conceptual diagrams and descriptions of how different sources (OCI, ConfigMap, inline YAML) and priorities would function.

The most direct "proof of concept" presented, albeit as a conceptual framework rather than a live execution, was the declarative Event Generator. The talk showed a YAML structure defining a test case, including:

• name: "Test Name"
• rule: "The_Target_Rule"
• context: Defining a process tree with specific args, capabilities, and names.
• resources: Describing entities like a client-server setup.
• steps: Outlining a sequence of syscalls to be triggered, simulating real kernel activity.

This YAML-based approach serves as a powerful proof that complex attack simulations and rule validations can now be defined declaratively. While not a live demo in the traditional sense, the detailed explanation of its structure and capabilities effectively demonstrated how users could leverage this tool to generate real kernel events and verify Falco's detection logic, making it a critical asset for rule developers and security testers. The underlying source code for these plugins and the operator is publicly available, allowing users to explore and implement these features independently.

Defensive Implications

▶ Watch: New audit log plugins for AKS and Vcloud MKS (8:10)

The advancements in Falco presented in this talk offer significant defensive implications for organizations operating in cloud-native environments. Adopting these new features can substantially bolster runtime security, streamline operations, and enhance compliance.

Firstly, upgrading Falco deployments to leverage the new minimal Wolfie container images is a critical first step. The reduction in image size and, more importantly, the drastic decrease in reported vulnerabilities (from 135 to 1) directly translates to a smaller attack surface and simplified Software Composition Analysis (SCA) compliance. This frees up security and DevOps teams from triaging numerous false positives, allowing them to focus on genuine threats. The 10% throughput improvement also means Falco can handle larger event volumes more efficiently, reducing the risk of missed detections due to performance bottlenecks.

Secondly, organizations utilizing AKS or Vcloud MKS should immediately integrate the new audit log plugins. This extends Falco's real-time threat detection capabilities beyond kernel events to cover critical control plane activities within these managed Kubernetes services. Monitoring audit logs for suspicious API calls, configuration changes, or access patterns provides an essential layer of defense against attacks targeting the Kubernetes control plane.

Thirdly, the Falco Operator is a game-changer for managing Falco at scale. Defenders should transition from Helm-based deployments to the operator to gain Kubernetes-native, declarative management of Falco instances and their artifacts. This facilitates multi-instance management, allowing for specialized Falco deployments—e.g., one DaemonSet focused on node-level syscalls and another Deployment monitoring specific cloud audit logs—each with tailored rule sets. The flexible artifact management with OCI registry support enables centralized, versioned control of security policies (rules, configs, plugins), ensuring consistency across multiple clusters and environments. The ability to use inline configurations and priorities empowers security teams to override default rule sets with organization-specific policies without modifying the original artifacts, promoting agility and maintainability.

Fourthly, for organizations focused on supply chain security, integrating Falco Actions into CI/CD pipelines (e.g., GitHub Actions) is highly recommended. By running Falco during build or test phases, developers can detect suspicious activities or policy violations before artifacts are deployed to production, thereby shifting security left and mitigating risks earlier in the development lifecycle.

Fifthly, the enhanced Falco Talon capabilities, particularly GCP actions and live captures, empower defenders with more robust automated response mechanisms. When a critical alert is triggered, Talon can automatically initiate pre-defined actions in GCP or capture live kernel activity for forensic analysis, enabling rapid containment and investigation.

Sixthly, the new container metadata plugin significantly improves the reliability and timeliness of security context. Defenders can now be confident that alerts, even from short-lived containers, will be instantly enriched with crucial metadata like container ID, image, and namespace. This precision is vital for accurate threat assessment and effective incident response. Security engineers should leverage the new Falco fields exposed by this plugin to write more granular and effective rules.

Finally, the declarative Event Generator provides an invaluable tool for proactive security testing. Defenders should utilize this to rigorously test their custom Falco rules against simulated attack scenarios, validate the effectiveness of Falco's detection engine, and ensure that their security policies are functioning as intended. This continuous validation process is crucial for maintaining a strong defensive posture against evolving threats.

Key Takeaways

• Streamlined Falco Images & Performance: Defaulting to minimal Wolfie images drastically reduces attack surface and SCA vulnerability reports (135 down to 1), while low-level optimizations (Clang, Zig, allocator changes) boost event throughput by 10%.
• Expanded Cloud-Native Monitoring: New plugins extend Falco's runtime security to AKS and Vcloud MKS audit logs, providing crucial visibility into control plane activities across more managed Kubernetes environments.
• Kubernetes-Native Management with Falco Operator: The new operator offers declarative, multi-instance management for Falco deployments and flexible artifact lifecycle control (rules, configs, plugins) via OCI registries, ConfigMaps, and inline overrides with priorities.
• Instant & Reliable Container Metadata: A new Go-based plugin ensures immediate and accurate container metadata enrichment for alerts by tapping into "container created" events, significantly improving security context and rule precision.
• Proactive Supply Chain Security & Response: Falco Actions integrate runtime security into CI/CD pipelines, while enhanced Falco Talon capabilities (GCP actions, live captures) provide robust automated response and forensic collection.
• Declarative Testing for Rules & Engine: The Event Generator's new YAML-based test suite allows for rigorously simulating real kernel activity and validating Falco's detection capabilities against complex, custom-defined scenarios.

About the Speaker(s)

The talk was presented by a team of dedicated Falco maintainers and contributors, demonstrating a collective expertise in cloud-native runtime security. Leonardo Grasso initiated the presentation, providing an overview of Falco and its mission as a graduated CNCF project. Luca Guerra detailed the latest improvements in Falco's core, focusing on image optimization and performance enhancements. Aurélie Vache introduced the new ecosystem plugins for AKS and Vcloud MKS audit logs, showcasing Falco's expanding integration capabilities. Jason Dellaluce delved into the technical intricacies of the new container metadata plugin, explaining its design and benefits. Finally, Aldo Lacuku presented the Falco Operator, outlining its role in Kubernetes-native management and flexible artifact handling, with Leonardo Grasso concluding with updates on the declarative Event Generator. Together, their contributions highlight the collaborative spirit and continuous innovation driving the Falco project forward.