System Introspection Applied to Micro-Services Pentests in K8S Clusters

System Introspection Applied to Micro-Services Pentests in K8S Clusters

Speakers: Laluka; Christophe Biguereau

Conference: GreHack

YouTube: https://www.youtube.com/watch?v=UStcX-fx0cw

Overview

In the rapidly evolving landscape of cloud-native applications, securing microservice architectures deployed on Kubernetes (K8s) clusters presents unique and formidable challenges for security professionals. This GreHack talk by Laluka and Christophe Biguereau, both from ManoMano, tackles this complexity head-on, introducing a novel approach to penetration testing that transcends traditional black, gray, and white-box methodologies. Their presentation, "System Introspection Applied to Micro-Services Pentests in K8S Clusters," argues that true visibility into system behavior, especially the "in-between world" of inter-process communication, is paramount for identifying subtle yet critical vulnerabilities that often go unnoticed.

The core of their work revolves around leveraging system introspection capabilities, specifically focusing on syscalls, to gain an unprecedented level of observability into how microservices interact within a Kubernetes environment. The speakers demonstrate how an open-source tool called Falco, traditionally used for runtime security monitoring, can be repurposed and enhanced to serve as a powerful offensive security instrument. This allows pentesters to trace the execution flow, file system operations, and network communications triggered by their exploits, even across multiple interconnected services, thereby uncovering hidden attack paths and validating assumptions in highly distributed systems.

This talk is particularly significant for several reasons. First, it addresses the growing pain point of security testing in large-scale microservice deployments, where manual inspection becomes impractical. Second, it champions a shift in pentesting mindset from static analysis or superficial interaction to deep, real-time behavioral analysis. Finally, by showcasing Falco's offensive capabilities, it empowers security teams to adopt sophisticated techniques for both red teaming and defensive monitoring, ultimately leading to more robust and resilient cloud infrastructure.

Background

▶ Watch: Understanding 'the in-between world' vulnerabilities (0:45)

The speakers introduce a critical concept they term the "in-between world" – a realm where individual components might appear secure in isolation, but when brought together, their interactions create fertile ground for vulnerabilities. They illustrate this with several compelling examples: a file upload server coupled with code execution capabilities, a system designed to drop keys coexisting with arbitrary file write access, or a website analysis tool (designed to fetch URLs) in a cloud environment leading to potential Server-Side Request Forgery (SSRF). These scenarios highlight that seemingly legitimate functionalities, when combined, can lead to devastating consequences, often due to unexpected communication patterns or partial inconsistencies between systems.

The challenge intensifies dramatically within modern microservice architectures, particularly those orchestrated by Kubernetes. As an application scales from a few monolithic services to hundreds of independent microservices (ManoMano, for instance, operates around 400), the complexity grows exponentially. Traditional pentesting approaches struggle here:

1. Black Box Testing provides no internal visibility, making it impossible to understand the ripple effects of a single request.
2. White Box Testing, while offering source code access, still doesn't reveal the runtime interactions and system calls that define the "in-between world."
3. Gray Box Testing (with some access) also falls short of providing the necessary depth.

The sheer volume of services makes it impossible for a small pentesting team (e.g., two pentesters for 400 services) to manually assess each component. Assumptions about how services interact become increasingly unreliable, leading to a significant blind spot where critical bugs can reside. This necessitates a solution that can provide deep introspection and observability into the operating system's core activities, regardless of the application's complexity or the tester's initial access level. The talk frames this as a fundamental shift required to move beyond surface-level security assessments in distributed systems.

Key Findings

▶ Watch: Introducing Falco for Kubernetes microservice introspection (10:00)

The central finding of this talk is that system introspection at the syscall level offers a powerful, often overlooked, avenue for identifying vulnerabilities in complex microservice environments, especially those built on Kubernetes. By monitoring the low-level interactions between processes and the operating system, pentesters can gain a "magic glasses" view into the "in-between world" where many critical flaws manifest. This approach overcomes the limitations of traditional black, gray, and white-box testing by providing real-time, granular visibility into what an application actually does on the host, rather than relying on assumptions or limited external views.

Specifically, the speakers demonstrate that leveraging Falco, an open-source runtime security tool, for offensive purposes transforms the pentesting process. Falco, which traditionally focuses on detecting anomalous behavior, can be configured to log all relevant syscalls related to file system operations, network interactions, and process execution. This capability allows pentesters to:

1. Trace execution flow: Understand exactly which files, libraries, and configurations are accessed during an attack, even if they span multiple microservices.
2. Identify hidden communication paths: Uncover internal network calls or inter-process communication that might not be obvious from external analysis.
3. Validate exploit efficacy: Confirm precisely what a successful exploit achieved (e.g., file write, code execution, sensitive data access) and its collateral effects.
4. Pinpoint vulnerable interactions: Detect attempts to open non-existent files in writeable paths, etc/password reads, or sudo executions that indicate potential privilege escalation or information disclosure.

This deep visibility significantly reduces the time and effort required for pentesting large infrastructures, enabling a small team to cover a much wider attack surface. It transforms the pentester from being "blind" to their actions into an informed operator, capable of understanding the full impact of their inputs and discovering vulnerabilities that would otherwise remain hidden within the intricate web of microservice interactions.

Technical Deep Dive

▶ Watch: Falco in action: live pentest demo in Kubernetes (16:00)

At the heart of system introspection lies the concept of syscalls (system calls). These are the fundamental interface between user-space applications and the operating system kernel. Whenever an application needs to perform a privileged operation—such as reading a file, opening a network socket, or spawning a new process—it must make a syscall. By monitoring these syscalls, one can observe the precise actions and interactions occurring within the system.

The talk explores various methods for hooking syscalls:

1. Userland hooking: Techniques like LD_PRELOAD or Frida, which inject code into user-space processes. While effective for specific applications, they don't provide system-wide visibility.
2. Kernel modules: Loading custom modules into the kernel to intercept syscalls. This offers deep access but carries significant risks, including system instability and potential crashes, especially in production environments.
3. eBPF (extended Berkeley Packet Filter): This is the preferred and most robust method. eBPF allows programs to run in the kernel without modifying the kernel's source code or loading new modules directly. It provides a safe, efficient, and flexible way to observe and filter syscalls, making it ideal for production systems and security monitoring. The speakers explicitly recommend "pick eBPF, don't crash your prod."

The inspiration for their approach comes from Sysdig (CD), a commercial tool that leverages eBPF (or kernel modules) to provide deep system visibility. However, for their offensive security needs and open-source principles, they pivoted to Falco. Falco is an open-source runtime security project, originally created by Sysdig and now a Cloud Native Computing Foundation (CNCF) project. It acts as a "security camera" for containerized environments, continuously monitoring syscalls and alerting on suspicious behavior based on a set of customizable rules.

Falco's architecture is divided into two main components:

• Falco Agent: This component is deployed on each host (Kubernetes node). It uses a kernel module or, preferably, eBPF probes to capture syscalls directly from the kernel. It then processes these events against a set of predefined and custom rules.
• Falco Sidekick: This is an optional but highly recommended component. It acts as an event router, receiving alerts from the Falco agents and forwarding them to various outputs like Slack, DataDog, databases, or custom webhooks. This allows for centralized logging, alerting, and analysis.

Additionally, optional components like a Falco UI and Redis can be used for better visualization and event storage.

Deploying Falco in a Kubernetes cluster requires careful orchestration:

• Falco Agent Deployment: Since the agent needs to run on every node to capture all syscalls, it is deployed as a DaemonSet. A DaemonSet ensures that a copy of the Falco agent pod runs on each node in the cluster.
• Falco Sidekick/UI Deployment: These components, being application services, are typically deployed as ReplicaSets to ensure high availability and scalability.
• Deployment Tooling: The speakers advocate using Helm, the package manager for Kubernetes, to manage the deployment of Falco. Helm charts define all the necessary Kubernetes resources (DaemonSets, ReplicaSets, ConfigMaps for configuration and rules, Services for access).
• Rule Management with Kustomize: A key insight for pentesting is the need to frequently update Falco rules without redeploying the entire service. To achieve this, they use Kustomize. Kustomize allows for declarative customization of Kubernetes manifests without templating. This enables splitting the service deployment from the rule deployment, allowing pentesters to create and update custom ConfigMaps containing their specific introspection rules more agilely.

For alert and event management, the captured syscall events are pushed to:

• Slack: For immediate, real-time alerts on critical findings during a pentest.
• DataDog: For long-term storage, historical analysis, and creating dashboards that visualize system behavior and potential attack paths. This provides a comprehensive overview of the infrastructure's activity and helps in creating statistics from the events, including Falco's own resource consumption (memory, CPU).

This integrated approach, combining eBPF-powered syscall capture with Kubernetes-native deployment and flexible rule management, transforms Falco from a mere security monitoring tool into a potent offensive security platform.

Demo / Proof of Concept

▶ Watch: Resource overhead, limitations, and challenges of Falco (20:30)

The live demonstration showcased the practical application of their Falco-based introspection tool, revealing its power and ease of use for pentesters. The setup involved a concise bash script, approximately 100 lines long, which orchestrated the entire process.

The demonstration began by executing this script, which first prompted the user to select the target Kubernetes cluster using fzf for fuzzy finding. The speaker wisely chose an "integration" cluster to avoid impacting production. Next, the script asked for a specific microservice to introspect, followed by a selection of Falco rules. For pentesting purposes, the speaker opted for a broad set of rules, essentially instructing Falco to log "pretty much any kind of event," directing the output to the standard output rather than DataDog or Slack.

After configuration, the system presented a multi-pane terminal interface:

1. Top Pane: Prompted for container selection within the chosen Kubernetes pod (a single pod can host multiple containers for the main application, secrets management, proxies, etc.). The main application container was selected.
2. First Main Pane (Top Left): Displayed raw Falco logs, useful for debugging the introspection setup itself.
3. Second Main Pane (Top Right): Offered a shell directly connected to the target container, allowing the pentester to execute commands.
4. Bottom Three Panes: Presented the parsed and beautified Falco logs, categorized into File System interactions, Network interactions, and Execution behaviors.

The demo proceeded with concrete examples:

• whoami command: Executing whoami in the shell immediately populated the "Execution behaviors" pane. More importantly, the "File System" pane revealed the intricate chain of syscalls: l.so loading, lib.so mapping, reading etc/password, and namespace switching. This showed that even a simple command involves a complex sequence of low-level operations, all visible through Falco. From an attacker's perspective, this is "gold" – a single line of input reveals a wealth of internal system behavior.
• ID command: Another basic command, ID, also registered in the execution logs, demonstrating the continuous monitoring of process activity.
• File Openings: The "File System" pane continuously updated with files being opened, read, and written, providing a real-time audit trail of application data access.
• curl ifconfig.me: To demonstrate network introspection, the curl ifconfig.me command was run. The "Network interactions" pane immediately displayed the HTTP GET requests to ifconfig.me, showing the source and destination IPs. This capability is invaluable for detecting SSRF (Server-Side Request Forgery) attempts or understanding cross-machine interactions. It functions "kind of like Wireshark, but more convenient" with built-in filtering.
• Non-existent File Access: A crucial part of the demo highlighted Falco's ability to detect attempts to open non-existent files. The speaker ran a custom script that tried to open a file named super_low that didn't exist. Falco logged these "file not found" events. This can be critical for identifying path traversal vulnerabilities or potential race conditions where an attacker might create a file in a writeable path that the application later attempts to open, leading to overwrite or code execution.
• Process, File, and Action Correlation: The demo emphasized that Falco logs provide not just what happened, but which process modified which file, when, and what actions led to it. This rich contextual information is invaluable for understanding complex attack chains.

In essence, the demo illustrated how a few lines of script and well-configured Falco rules could grant "superpowers" to pentesters in Kubernetes, providing deep, actionable insights into system behavior that is otherwise opaque.

Defensive Implications

▶ Watch: Key benefits and conclusion for pentesters and companies (26:30)

While primarily presented from an offensive perspective, the insights gleaned from this talk have profound defensive implications for organizations operating microservices on Kubernetes. Falco, being a runtime security tool, is inherently designed for defense, and its capabilities highlighted in the talk underscore its value as a foundational component of a robust security posture.

1. Enhanced Runtime Security Monitoring (SOC in a Box): Falco can effectively serve as a "small yet efficient SOC" for cloud-native environments. By deploying Falco agents across all Kubernetes nodes and configuring custom rules, defenders can monitor for a wide array of suspicious activities. This includes detecting unauthorized file modifications, unexpected process executions (e.g., sudo with unusual arguments, php files being created or executed in non-standard locations), web shells, or unusual network connections (e.g., unexpected socket opens to external IPs). The integration with Slack provides immediate alerts, while DataDog provides historical data for incident analysis and trend spotting.

1. Proactive Anomaly Detection and Threat Hunting: Beyond generic rules, defenders can create custom Falco rules tailored to their specific applications and environments. By understanding normal behavior, any deviation can trigger an alert. For example, if a specific microservice should never open a raw network socket or write to /tmp, a rule can be created to flag such events. This proactive approach helps in identifying zero-day exploits or novel attack techniques that might bypass traditional perimeter defenses.

1. Migration and Compliance Validation: The speakers shared a real-world example: using Falco to monitor the migration from AWS IMDS V1 to IMDS V2. By creating rules to detect calls to the older V1 endpoint, they could track which services were still using the deprecated version, ensuring compliance and successful migration. This demonstrates Falco's utility beyond pure security, extending to operational and compliance monitoring.

1. Incident Response and Forensics: The historical data collected by Falco (e.g., in DataDog) is invaluable during incident response. When a security incident occurs, having a detailed timeline of all syscalls, file operations, and network events leading up to and during the incident allows security teams to:

• Reproduce the incident: Understand the attacker's steps and the system's response.
• Determine scope and impact: Identify all affected systems and data.
• Improve future defenses: Learn from the attack chain and implement specific rules or controls to prevent recurrence.

1. Challenges and Considerations for Defenders: The talk also candidly addresses the costs and limitations of running Falco in production:

• Resource Overhead: Falco agents consume CPU, memory, network, and storage resources. Careful resource allocation and tuning are essential to avoid performance impacts on applications.
• Rule Tuning and False Positives: Generic rules often lead to a high volume of false positives. Defenders must invest time in customizing rules to match their environment's baseline behavior, which is a continuous effort. Untuned rules can also create alert loops (Falco logging its own logs, which triggers more alerts).
• Deployment Complexity: While Helm simplifies deployment, integrating Falco into existing monitoring, logging, and security workflows requires significant effort.
• Kernel Compatibility: Older kernels (pre-5.8) may require more privileged access or external modules for eBPF, adding complexity and potential risk.
• Limited Scope: Falco only sees syscalls on the nodes where its agents run. It won't directly monitor external databases or services outside the Kubernetes cluster, though it will see the interactions with those services.

Despite these challenges, the ability to gain deep, low-level insights into runtime behavior makes Falco an indispensable tool for defenders seeking to secure complex, dynamic Kubernetes environments. It empowers them to move beyond reactive security to a more proactive, behavior-based defense strategy.

Key Takeaways

• System Introspection is Critical for K8s Pentesting: Traditional black/gray/white box approaches are insufficient for securing complex microservice architectures due to the "in-between world" of inter-process communication. Deep system visibility via syscalls is essential.
• Falco as an Offensive and Defensive Powerhouse: The open-source runtime security tool Falco, leveraging eBPF, can be repurposed by pentesters to gain unprecedented visibility into file system, network, and execution behaviors within Kubernetes clusters. It also serves as a robust defensive monitoring tool.
• eBPF is the Preferred Method for Syscall Hooking: eBPF offers a safe, efficient, and flexible way to observe kernel events without risking system stability, making it superior to userland hooks or kernel modules for production environments.
• Granular Visibility Uncovers Hidden Vulnerabilities: By logging and parsing syscalls, pentesters can trace entire attack chains, identify subtle interaction flaws (e.g., non-existent file writes, unexpected network calls), and validate exploit effects across multiple microservices.
• Kubernetes-Native Deployment and Rule Management: Falco can be effectively deployed in Kubernetes using DaemonSets (for agents) and Helm for orchestration, with Kustomize enabling agile, independent management of custom security rules.
• Requires Careful Tuning and Resource Management: While powerful, Falco introduces resource overhead (CPU, memory, storage) and demands continuous effort in tuning rules to minimize false positives and avoid performance degradation.

About the Speaker(s)

Laluka and Christophe Biguereau are security professionals who demonstrated their expertise in offensive security, particularly within cloud-native environments. Both speakers work at ManoMano, an e-commerce platform, where they are involved in pentesting and securing the company's extensive microservice infrastructure. Their work focuses on the practical challenges of securing large-scale Kubernetes deployments and developing innovative approaches to overcome the limitations of traditional security assessments. Their presentation reflects a deep understanding of both offensive techniques and the underlying technical architecture of modern cloud applications.