Detection And Blocking With BPF Via YAML

Kev Sheldrake (Security Software Developer and Researcher)

TROOPERS23 · Day 2 · Defense & Management · Defense & Management

Overview

In this insightful talk from TROOPERS, Kev Sheldrake, a Security Software Developer and Researcher, presents a compelling argument for a paradigm shift in how organizations approach logging and security detection. Moving beyond the limitations of traditional, rigid logging systems and EDRs, Sheldrake introduces Tetragon, an open-source tool developed by Isovalent that harnesses the power of eBPF (extended Berkeley Packet Filter). The core premise is to empower defenders to "build their own logging system" by leveraging eBPF's kernel-level visibility and control, configured entirely through simple YAML files, eliminating the need for complex BPF programming.

Watch on YouTube · Slides

Visual summary for Detection And Blocking With BPF Via YAML by Kev Sheldrake
Visual summary for Detection And Blocking With BPF Via YAML by Kev Sheldrake

Key moments

  1. 0:00 Introduction and "Build Your Own Logging System"
  2. 4:25 BPF power, portability, and challenges
  3. 8:20 Introducing Tetragon: YAML-driven BPF security
  4. 11:40 Tetragon's detection, blocking, and killing capabilities
  5. 25:50 Live Demo: Blocking symlink to /etc/passwd
  6. 35:00 Advanced file descriptor tracking and path resolution
  7. 44:00 Tetragon's future roadmap and community plans
  8. 49:00 Q&A: Protecting Tetragon from tampering and BPF rootkits

Detection And Blocking With BPF Via YAML

Speakers: Kev Sheldrake, Security Software Developer and Researcher

Conference: TROOPERS

YouTube: https://www.youtube.com/watch?v=ZoVSWo6OG6o

Overview

In this insightful talk from TROOPERS, Kev Sheldrake, a Security Software Developer and Researcher, presents a compelling argument for a paradigm shift in how organizations approach logging and security detection. Moving beyond the limitations of traditional, rigid logging systems and EDRs, Sheldrake introduces Tetragon, an open-source tool developed by Isovalent that harnesses the power of eBPF (extended Berkeley Packet Filter). The core premise is to empower defenders to "build their own logging system" by leveraging eBPF's kernel-level visibility and control, configured entirely through simple YAML files, eliminating the need for complex BPF programming.

Sheldrake posits that the ever-evolving landscape of cyber threats, characterized by constant change and novel attack techniques, renders static logging solutions inherently insufficient. If even security experts cannot predict the next zero-day, how can pre-packaged logging systems possibly detect them? Tetragon aims to bridge this gap by providing unparalleled flexibility and deep kernel introspection, enabling real-time detection, alerting, and even proactive blocking and killing of malicious processes directly from the kernel. This approach offers a powerful, adaptable defense mechanism that can evolve daily to counter emerging threats, a capability often lacking in commercial and even some open-source security tools.

Background

▶ Watch: Introduction and "Build Your Own Logging System" (0:00)

The genesis of Tetragon stems from a fundamental dissatisfaction with the status quo of security logging and detection. Sheldrake highlights that most logging systems, whether open-source like K-audit (and its user-space component Auditd) or proprietary EDRs, offer severely limited configurability. Users are often restricted to predefined options, unable to truly customize what gets logged or how events are interpreted. While open-source solutions theoretically allow code modification, the inherent risks of altering kernel code and the steep learning curve make this impractical for most organizations. Proprietary EDRs, on the other hand, intentionally obscure their internal workings, providing only a superficial layer of configuration.

This rigidity creates a critical vulnerability: if logging systems cannot adapt to new, unforeseen attack techniques, they will inevitably fail to detect them. Sheldrake argues that waiting for vendor updates for every new exploitation method is a losing battle. The solution, he contends, is to build a logging system that can change as rapidly as the threat landscape itself. This is where eBPF enters the picture. eBPF is a revolutionary kernel technology that allows users to run custom, sandboxed programs within the Linux kernel, hooking into virtually any kernel function, network operation, or syscall. Its power lies in its ability to observe and manipulate kernel events with minimal overhead and without requiring kernel module compilation, making it highly portable across different kernel versions (e.g., from 4.19 to modern 6.x branches).

However, Sheldrake, having previously ported Sysmon from Windows to Linux using eBPF, acknowledges a significant barrier to entry: eBPF is "really, really difficult." Developing eBPF programs typically involves writing complex C code, understanding intricate kernel data structures, and navigating the stringent BPF verifier. The verifier performs static analysis to ensure programs are safe, terminate within a million instructions, don't dereference null pointers, and only use approved helper functions, effectively preventing kernel crashes but adding considerable development complexity. This high barrier means that while eBPF is powerful, its full potential remains largely inaccessible to most security teams, making a generic, easily configurable abstraction layer highly desirable.

Key Findings

▶ Watch: Introducing Tetragon: YAML-driven BPF security (8:20)

The central contribution of this talk is Tetragon, an open-source tool designed to democratize access to eBPF's power for security monitoring and enforcement. Available on GitHub (github.com/cilium/tetragon), Tetragon's key innovation is its reliance on YAML for configuration, removing the need for any direct eBPF programming. This declarative approach allows anyone familiar with YAML to define complex security policies, enabling rapid adaptation to new threats.

Tetragon provides several critical capabilities:

  • Intelligent Data Type Handling: Unlike other logging systems that might output raw pointers, Tetragon understands kernel data structures (e.g., file descriptors, sockets, SKBs – socket buffers). When it captures these, it automatically resolves and prints their meaningful contents in JSON output, making logs immediately actionable.
  • Context-Aware Event Enrichment: Beyond function arguments, Tetragon can access and integrate extensive contextual information about processes, including PID, UID, the executing binary, container details (pods, namespaces), and capabilities. This enriches every event, providing a comprehensive "full history" of the process without requiring post-log correlation in a SIEM.
  • Container Awareness: A significant advantage over traditional K-audit-based solutions (like Elastic's Auditbeat), Tetragon is inherently container-aware. It can distinguish between host-level PIDs and container-specific PIDs, providing the correct context for containerized workloads, which is crucial in Kubernetes environments.
  • Flexible Hooking: Users can specify where to hook the kernel—syscalls, kernel functions, network operations, or tracepoints—and define what data to observe and what actions to take. This includes hooking on function return values, which is essential for capturing post-execution state (e.g., the content of a buffer after a read operation).
  • Proactive Enforcement: Tetragon transcends mere detection. It can be configured to block specific function calls by returning an error code (e.g., EPERM) or to kill processes outright from within the kernel. This is a critical distinction from many EDRs that kill processes from user space, often after a malicious action has already completed.
  • Canary Integration: For high-fidelity alerts, Tetragon can trigger canaries (e.g., Thinkst Canaries, Canarytokens), sending immediate notifications (email, SMS, Slack) when highly suspicious activity is detected.
  • Community-Driven Detections: Sheldrake envisions Tetragon fostering a community similar to Sysmon for Windows, where users share and contribute a repository of detection rules, continuously enhancing the collective defense posture.

Technical Deep Dive

▶ Watch: Live Demo: Blocking symlink to /etc/passwd (25:50)

Tetragon’s power lies in its ability to abstract the complexity of eBPF into simple YAML configurations, allowing users to define tracing policies that specify where to hook, what to observe, and what actions to perform.

YAML Configuration:

A typical Tetragon YAML policy defines a kprobe (kernel probe) or tracepoint hook. For instance, to monitor the clock_settime syscall, a policy might specify:

This YAML instructs Tetragon to attach an eBPF program to the sys_clock_settime syscall, capture the which_clock argument (at index 0), ignore calls from PIDs 0 and 1 (even if containerized), and generate a detection event.

Finding Kernel Hooks:

Identifying the correct kernel functions or syscalls to hook is crucial. Sheldrake recommends elixir.bootlin.com as an invaluable resource for navigating kernel source code. For syscalls, one typically searches for sys_ or do_sys_ followed by the syscall name. A key technical detail highlighted is to hook after the kernel has copied user-space buffers into kernel memory. This prevents phantom attacks (as demonstrated at Defcon 29), where an attacker modifies a user-space buffer after a logging system has observed it but before the kernel acts upon it, leading to misreported logs.

Advanced Tracking Mechanisms:

  1. File Descriptor (FD) Tracking (follow_fd):
  • Problem: Matching on file paths using string comparisons is inherently unreliable due to directory traversal, relative paths, and current working directories. A user-provided path like ../../etc/password or just password (if in /etc) is difficult to normalize and match accurately.
  • Tetragon's Solution: Tetragon resolves the actual canonical path by tracing kernel internals. It hooks the FD_install kernel function, which is called whenever a new file descriptor is created (e.g., by open, create). When FD_install provides a file descriptor and a pointer to a struct file, Tetragon stores this mapping (FD to resolved absolute path) in a kernel-resident hash map. It also hooks close syscalls to remove FDs from this map when they are no longer active.
  • This allows subsequent operations (like write to an FD) to be matched against the true file path, not the user-provided string. For example, a policy could block writes to /etc/password by hooking sys_write and matching index 0 (the file descriptor) against the resolved path /etc/password.
  1. Socket Tracking (track_sock):
  • Problem: Network operations are often asynchronous. A connect syscall, for instance, sends a SYN packet, but the kernel might handle the subsequent SYN-ACK and connection establishment in a different process context later. This makes it challenging to attribute network events to the originating process.
  • Tetragon's Solution: Tetragon tracks socket creation, which is typically a synchronous operation. It hooks the SK_alloc kernel function (and tcp_create_openreq_child for TCP-specific cases) to map a unique struct sock* pointer (representing the socket in kernel memory) to the process that created it. This mapping is stored in an in-kernel database.
  • When subsequent asynchronous network events occur (e.g., a SYN-ACK arriving for an existing socket), Tetragon can look up the struct sock* in its database, identify the original process, and enrich the event with the correct process ID.
  • This enables tracking TCP connects, listens (by hooking the listen k-probe), and accepts (by monitoring tcp_set_state transitions from SYN_RECV to ESTABLISHED).
  1. UDP Tracking with Rate Limiting:
  • Problem: UDP is connectionless; a single socket can send datagrams to many endpoints. Logging every datagram would generate excessive volume, potentially overwhelming the eBPF ring buffer and log storage.
  • Tetragon's Solution: Similar to how Sysmon handles UDP, Tetragon implements rate limiting. It can be configured to report only unique datagrams within a specified time window (e.g., 30 minutes). This effectively captures the "establishment" of a UDP communication channel without flooding logs with redundant data, providing a balance between visibility and performance.

Future Development:

Tetragon is under active development. Upcoming features include IPv6 support for socket tracking, more comprehensive file operations (extracted from Isovalent's Enterprise version), and a more user-friendly interface for tracing policies. This includes allowing users to define high-level policies like "log all connects," which Tetragon would then translate into the necessary complex YAML and eBPF hooks.

Demo / Proof of Concept

▶ Watch: Advanced file descriptor tracking and path resolution (35:00)

Kev Sheldrake demonstrated Tetragon's capabilities with several live examples, showcasing its flexibility in detection, alerting, blocking, and killing.

  1. Detecting symlinkat to /etc/password:

The first demo involved a simple YAML policy designed to detect attempts to create a symbolic link to /etc/password. The policy hooked the symlinkat syscall and matched its argument (the target path) to ensure it ended with "password". When ln -s /etc/password p was executed, Tetragon successfully generated a JSON event in user space. The event clearly showed the ln process, its bash parent, the symlinkat function call, and the argument /etc/password, confirming the detection.

  1. Alerting with Canaries:

Building on the previous example, Sheldrake modified the YAML policy to include an action: dns_lookup. This configuration integrates with Canarytokens or Thinkst Canaries, which trigger an external alert (e.g., a DNS query to a specific domain) when a predefined event occurs. Upon re-executing the symlinkat command, Tetragon not only generated the detection event but also triggered an immediate alert. Sheldrake showed screenshots of this alert appearing in his Slack channel and via email, demonstrating real-time, high-fidelity notification of critical security events.

  1. Blocking symlinkat:

To showcase proactive enforcement, the YAML policy was further modified to include action: override with a return_value: -1. This tells Tetragon to intercept the symlinkat syscall and return an error code (typically EPERM for "Operation not permitted") instead of allowing the kernel function to execute. When the ln -s /etc/password p command was attempted, the terminal immediately reported "Operation not permitted," and the symbolic link was not created. This demonstrated Tetragon's ability to prevent malicious actions at the kernel level.

  1. Killing Process on symlinkat:

The final live demo extended the blocking capability by adding action: kill to the policy. This instructed Tetragon to not only block the symlinkat syscall but also to immediately terminate the offending process. When the ln command was executed, the process was instantly "Killed," illustrating Tetragon's capability to neutralize threats completely and prevent any further actions by the compromised process.

Sheldrake also described a non-live demo of blocking writes to /etc/password. This policy would hook the sys_write syscall, leveraging Tetragon's file descriptor tracking to match against the resolved absolute path /etc/password (not just a string). If a process attempted to write to this file, Tetragon would block the write and kill the process, demonstrating the power of precise, path-aware enforcement.

Defensive Implications

▶ Watch: Q&A: Protecting Tetragon from tampering and BPF rootkits (49:00)

Tetragon offers profound defensive implications, fundamentally changing how security teams can monitor and protect Linux environments. Its kernel-level capabilities, combined with user-friendly YAML configuration, empower defenders in several key ways:

  1. Proactive Threat Neutralization: The ability to block or kill processes directly from the kernel, before a malicious action completes, is a game-changer. Unlike many EDRs that react from user space (often after data exfiltration or system damage has occurred), Tetragon intervenes at the earliest possible point, preventing impact. This is crucial for stopping ransomware, rootkits, and other high-impact threats in their tracks.
  1. Adaptive Security Posture: The YAML-driven configuration allows security teams to rapidly develop and deploy custom detection and enforcement policies in response to new CVEs, zero-days, or emerging attack techniques. This agility eliminates the dependency on vendor updates and enables organizations to "build their own logging system" that evolves daily, as advocated by Sheldrake.
  1. Unparalleled Visibility and Context: By hooking deep within the kernel and intelligently resolving complex data structures (like file descriptors and sockets), Tetragon provides a level of visibility unmatched by many traditional tools. Defenders gain precise insights into process execution, parent-child relationships, container context (PID, namespace), and the true nature of file and network operations (canonical paths, mapped sockets), facilitating faster and more accurate incident response.
  1. Enhanced Container Security: Tetragon's native container awareness is critical for modern cloud-native environments. It correctly attributes events to specific containers and their internal PIDs, overcoming a significant limitation of K-audit-based solutions that struggle with container context. This enables granular security policies tailored to individual container workloads.
  1. Building Custom Security Controls: Tetragon provides the primitives necessary to build highly customized security controls, from application whitelisting (e.g., allowing only specific binaries to call certain syscalls) to micro-segmentation (e.g., restricting network connections based on process identity within a container). It can function as a highly flexible firewall, IDS/IPS, or sandboxing mechanism at the kernel level.
  1. BPF Integrity Monitoring: Given the rise of BPF-based rootkits and other malicious eBPF programs, Tetragon's ability to monitor eBPF syscalls and kernel functions (like BPF_check, security_perf_event_alloc, security_bpf_map_alloc) is a vital defensive measure. This allows defenders to detect unauthorized loading of eBPF programs or manipulation of eBPF maps, providing early warning of sophisticated kernel-level attacks.
  1. High-Fidelity Alerting: Integration with canary tokens ensures that critical detections immediately trigger high-priority alerts, cutting through log noise and directing security teams to the most pressing threats.

In essence, Tetragon transforms eBPF from a complex kernel technology into an accessible and powerful tool for defenders, enabling a proactive, adaptable, and deeply insightful security posture against the constantly evolving threat landscape.

Key Takeaways

  • Traditional logging systems are often too rigid and slow to adapt to new, evolving attack techniques, leaving organizations vulnerable to zero-day exploits.
  • Tetragon empowers defenders to build highly customized, kernel-level detection and blocking capabilities using eBPF, configured through simple YAML, without requiring complex BPF programming expertise.
  • The tool provides unparalleled visibility into kernel operations, intelligently resolving complex data structures like file descriptors and sockets to provide accurate, context-rich event logs (e.g., canonical file paths, process-mapped network connections).
  • Tetragon enables real-time, proactive enforcement by allowing direct blocking of syscalls/kernel functions or immediate killing of malicious processes from within the kernel, preventing actions before they complete.
  • Its native container awareness makes it a superior choice for securing modern cloud-native and Kubernetes environments, addressing limitations of traditional host-based logging solutions.
  • By fostering a community-driven repository of detection policies, Tetragon aims to continuously enhance collective defense capabilities against emerging threats.

About the Speaker(s)

Kev Sheldrake is a distinguished Security Software Developer and Researcher with a strong focus on kernel-level security and eBPF technology. He is known for his significant contributions to the security community, notably his work at Sysinternals where he successfully ported the renowned Sysmon tool from Windows to Linux. This endeavor leveraged eBPF to replace the Windows driver, making a powerful security monitoring tool available to the Linux ecosystem as open source. Currently, Kev works as a Software Engineer at Isovalent, the company behind the open-source Cilium project and Tetragon, where he continues to innovate in the field of eBPF-driven security. His passion lies in making advanced kernel capabilities accessible to defenders, enabling them to build more resilient and adaptive security systems.

Reviews

Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT

Sheldrake drops the mic on why legacy Auditd and black-box EDRs are dead ends for Linux threat hunting. Tetragon bypasses the nightmare of the BPF verifier, handing defenders a YAML-driven scalpel to hook kernel functions, resolve canonical paths, and kill processes before user-space even blinks. If your defense relies on parsing raw syscalls after the fact, you've already lost.

Heather Calloway (CISO) — SOLID

Sheldrake outlines how Tetragon uses YAML to abstract the complexity of eBPF, giving defenders kernel-level visibility and blocking without requiring specialized C developers. It shifts custom detection from reactive user-space logging to proactive kernel enforcement. I’d have my detection engineering lead evaluate this immediately for our Kubernetes fleet.

→ Top-rated talks at TROOPERS23

All talks from TROOPERS23