Reinventing ETL for Detection and Response Teams

Reinventing ETL for Detection and Response Teams

Speakers: Josh Liburdi

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=im5xfAHeiRo

Overview

In his BSidesSF 2024 talk, "Reinventing ETL for Detection and Response Teams," Josh Liburdi tackles a topic he admits is often considered "boring" but is, in his view, critically important: Extract, Transform, Load (ETL) processes in cybersecurity. Liburdi argues that the current state of security ETL is fundamentally broken, leading to significant challenges for Security Operations Center (SOC), Incident Response (IR), and Threat Hunt teams. His presentation is a call to action to move beyond simply collecting data and instead focus on transforming raw data into actionable information and knowledge in real-time.

Liburdi highlights that despite millions spent on security platforms, the data presented to analysts is often uncontextualized, requiring them to write complex, multi-line queries to derive basic insights. This inefficiency leads to analyst fatigue, potential inaccuracies, and a reliance on deep domain expertise that is unsustainable, especially during high-pressure incidents like ransomware intrusions. The core of his talk is the introduction of innovative, real-time data contextualization patterns—dubbed "Time Travel," "Telephone," and "NXR"—implemented within the data pipeline, rather than at the SIEM or analyst level.

The talk is highly technical, offering concrete architectural patterns and examples of how these concepts are applied in production using an open-source project called Substation. Liburdi's vision is to make security data effortlessly insightful, empowering detection and response teams with enriched, decision-ready information, thereby "inventing" a form of security ETL that, he contends, has never truly existed.

Background

▶ Watch: The core problem: SOC, IR, Hunt teams use siloed, bad data. (01:00)

ETL, an acronym for Extract, Transform, Load, describes the general process of moving data from one system to another, typically involving extraction from source systems, transformation into a suitable format, and loading into a destination like a Security Information and Event Management (SIEM) system, data lake, or data warehouse. While seemingly straightforward, Liburdi asserts that the current implementation of ETL within the security industry is "quite bad," failing to meet the needs of modern detection and response teams.

The problem stems from several critical deficiencies. Firstly, security vendors, non-security vendors, and even internal engineering teams often provide raw, uncontextualized data. This "bad data" is then ingested into SIEMs or data lakes, which, according to Liburdi, are "actually quite bad at the for the reason you bought it which is analysis." The consequence is that the burden of making sense of this data falls squarely on the security analysts. They are forced to possess extensive domain expertise and spend inordinate amounts of time crafting complex queries—sometimes 40 or 50 lines long—to piece together the necessary context for alert triage, investigations, or threat hunting. This process is not only inefficient but also prone to errors, especially under the stress of a 2 AM ransomware incident.

A significant, yet often overlooked, aspect of this problem is data decay. Liburdi explains data decay as the phenomenon where data, accurate at the time of observation, becomes outdated over time through no fault of the consumer. A simple example is a phone number that changes, rendering previous contact information obsolete. In a security context, this is highly prevalent with IP address analysis, where geolocation or attribution data applied days or weeks after an event may no longer be accurate due to IP reassignments or changes in proxy networks. The speaker highlights the impressive scale of this problem with services like the Luminati proxy, which boasts over 8 million active IPs with an 11% daily churn rate. Data decay also affects DNS domains, web pages, cloud metadata, files, and even abstract concepts like threat intelligence. Relying on SIEMs to perform enrichment functions on historical data without accounting for its temporal validity leads to potentially inaccurate conclusions.

Liburdi also critiques current industry trends. He expresses confusion and skepticism regarding the concept of "Federated SIEM," primarily due to concerns about data governance and the reliability of data managed by external vendors. In contrast, he strongly advocates for unified data models, stating they are "the standard" and that any team not using them is "deficient" or "malnourished." His ultimate goal for security data is for it to be "effortless to derive insight from data," a benchmark he believes "pretty much every security data vendor fails at." This foundational critique sets the stage for his proposed "reinvention" of security ETL, moving beyond mere collection to active, real-time contextualization.

Key Findings

▶ Watch: Introduction of 'data decay' as a critical, overlooked problem in security data. (08:00)

Liburdi's talk identifies several key findings and principles for effective security ETL, challenging conventional approaches and proposing a new paradigm for detection and response teams:

• The Current State of Security ETL is Inadequate: Existing security ETL processes are "quite bad," characterized by raw, uncontextualized data, overly complex SIEM queries, and platforms that fail to provide effortless insight. This leads to analyst fatigue, reliance on deep domain expertise, and a high potential for inaccurate conclusions.
• Good Security ETL is More Than Collection: Effective ETL must go beyond simply collecting data. It involves contextualizing data, making it actionable, and enabling the effortless derivation of insight. These steps transform raw data into valuable information and knowledge.
• Data Decay is a Pervasive Threat to Analysis: A critical, often unaddressed problem is data decay, where data attributes (e.g., IP geolocation, DNS records, cloud metadata, user roles) become outdated over time. This undermines the accuracy of historical analysis and requires real-time solutions.
• Unified Data Models are Essential: Liburdi asserts that unified data models are no longer optional but "table stakes" for modern detection and response teams. They standardize data representation, making it easier to analyze and share across different security functions.
• The Data Pipeline is the Optimal Location for Enrichment: While enrichment can occur at various stages, the data pipeline is identified as the best place. This is the first opportunity to exert influence over the data, typically within seconds to minutes of observation, effectively mitigating data decay by enriching data before it reaches the SIEM.
• Vendor "Context" is Often Insufficient: Commercial security vendors frequently offer limited or outdated forms of context, such as CSV uploads or daily scans of cloud accounts. True real-time, dynamic contextualization is rarely provided.
• Three Novel ETL Patterns for Real-time Contextualization: Liburdi introduces three specific architectural patterns, implemented in the data pipeline, that enable advanced real-time data enrichment:

1. Time Travel Pattern: Allows data to enrich itself by leveraging a fast-reading enrichment function that stores high-latency context (e.g., DNS resolution, external API calls) in a database, which a slower-reading transform function then uses to enrich events before they are sent to the SIEM. This is particularly effective for building process lineage.
2. Telephone Pattern: Facilitates real-time information sharing across multiple disparate data streams (e.g., EDR, identity platform, device management) to build a comprehensive context for events, such as linking an EDR event to a specific user, their role, and account status.
3. NXR Pattern: Involves moving traditional SIEM queries into the data pipeline to generate "threat information" as data. This transforms alerts into searchable, contextualized data points, breaking down knowledge silos and providing analytical signposts with descriptions, MITRE ATT&CK mappings, references, and flexible risk scores.

• Open-Source Solutions Offer Powerful Alternatives: Projects like Substation demonstrate that these advanced ETL capabilities can be built and deployed at a fraction of the cost of commercial vendor solutions, especially when leveraging serverless architectures.

Technical Deep Dive

▶ Watch: First novel ETL pattern: 'Time Travel' for self-enrichment (e.g., EDR process... (16:00)

The core of Liburdi's technical deep dive revolves around the implementation of advanced ETL patterns within the data pipeline, primarily using Substation, an open-source project developed and used in production at Brex for over three years. Substation, open-source for over two years, is presented as a cost-effective alternative to commercial solutions, particularly for organizations already leveraging cloud platforms like AWS.

Liburdi outlines three primary use cases for such a platform:

1. Routing Data: The most basic use case, involving moving data between systems. While easy to achieve, he advises against paying vendors for this if an organization is already in AWS, as open-source tools like Substation can handle it at a much lower cost.
2. Normalizing Data to a Data Model: This is deemed "table stakes" for modern detection and response teams. It involves transforming disparate data sources into a unified, standardized format, which is crucial for consistent analysis and correlation.
3. Enriching Data to Generate Context: This is the most compelling and challenging use case, focusing on real-time data contextualization. Liburdi criticizes vendors for offering inadequate "context" (e.g., CSV uploads, daily AWS account scans) and emphasizes the need for dynamic, up-to-date enrichment.

The proposed solutions leverage a serverless architecture, primarily within AWS, though Liburdi notes that equivalent technologies (e.g., Kafka instead of Kinesis, Kubernetes instead of Lambda) can be substituted. This approach minimizes operational overhead for small teams by offloading server management to the cloud provider.

The Time Travel Pattern

This pattern addresses the challenge of enriching data with information that might have a higher latency to acquire (e.g., external API calls, DNS resolution) but is crucial for immediate context.

• Mechanism: A data stream (e.g., from an Endpoint Detection and Response (EDR) system) is consumed by two functions at different frequencies.
• An enrichment function/consumer reads data very quickly. Its sole purpose is to perform high-latency data enrichment (e.g., resolving DNS, calling external APIs, making data "pretty") and store this context in a database (e.g., DynamoDB for high throughput).
• A transform function/consumer reads the data stream at a slightly higher latency (e.g., 10 seconds later). By the time it receives an event, the necessary high-latency context for that event has already been processed and stored in the database by the enrichment function. The transform function then queries the database, retrieves the pre-enriched context, inserts it into the event, applies the unified data model, and sends the fully contextualized event to the SIEM.
• Example: An EDR event might only contain a process ID and a generic name like "Spotify." The time travel pattern allows the enrichment function to resolve the process ID to its full command line, start time, and parent process ID. This process is recursive, building a complete process lineage (e.g., Spotify spawned by board D, which was spawned by launchd on macOS), providing deep context without complex SIEM joins.

The Telephone Pattern

This pattern focuses on sharing and combining knowledge from multiple, disparate data sources in real-time to build a richer context for events.

• Mechanism: Multiple data streams (e.g., EDR, identity platform, device management, cloud services) are integrated. Information from one stream is used to enrich events from another, leveraging a shared database for lookups.
• Example: An EDR event might lack a hostname. The time travel pattern (or a similar real-time lookup) first inserts the hostname. With the hostname, the telephone pattern then queries a device inventory service to identify the assigned user (e.g., "Alice"). Further, using Alice's email, it queries an Identity Provider (IDP) data stream to retrieve her role (e.g., "manager in engineering organization") and real-time account status (active, suspended, deactivated). This comprehensive user and device context is embedded directly into the EDR event, making it searchable by user, not just hostnames or IDs.

The NXR Pattern

This pattern proposes moving traditional SIEM detection logic and queries directly into the data pipeline, transforming "threat information" into data rather than just alerts.

• Mechanism: Instead of writing complex queries in the SIEM to identify specific threat behaviors, these queries are executed within the data pipeline. When a condition is met, the pipeline generates a new, enriched event that represents this "threat information."
• Example: Detecting the securityd_authtrampoline process on macOS. The NXR pattern would enrich this event with a human-readable description ("an authentication prompt that is called by an API that says execute with privileges"), map it to MITRE ATT&CK tactics and techniques, include references to relevant external resources (e.g., "Objective-C blog"), and assign a flexible risk score. This "threat information" is then ingested into the SIEM as regular data, making it universally searchable and accessible across SOC, IR, and hunt teams, breaking down knowledge silos. The flexible risk scoring allows for calculating risk not just for hosts or users, but for processes, internet servers, or even entire teams.

For handling significant throughput (terabytes of logs per day) on context tables, Liburdi explicitly recommends DynamoDB due to its scalability. The use of serverless services like AWS Lambda is key to managing the complexity of multi-layered enrichment without requiring a large team to manage underlying infrastructure.

Demo / Proof of Concept

▶ Watch: Second novel ETL pattern: 'Telephone' for cross-stream contextualization (e.g... (20:00)

While the talk did not feature a live coding demonstration, Josh Liburdi effectively presented the "proof of concept" through visual examples of data transformation. For each of the three ETL patterns—Time Travel, Telephone, and NXR—he displayed side-by-side comparisons of "before processing" and "after processing" data. These visual aids clearly illustrated the profound impact of real-time contextualization on raw security events.

For the Time Travel Pattern, the demonstration showed a raw EDR event, which typically contains minimal information like a process ID and a generic name (e.g., "Spotify"). The "after processing" example revealed an event dramatically enriched with a full process lineage. This included the complete command line of the executed process, its start time, and crucially, the same detailed context for its parent process (e.g., board D) and grandparent process (e.g., launchd on macOS). This visual transformation highlighted how a single event could encapsulate a deep, multi-layered understanding of process execution, eliminating the need for complex, multi-join queries in the SIEM.

The Telephone Pattern was demonstrated with an EDR event that initially lacked critical host and user context. The "after processing" view showcased how the event was enriched first with the correct hostname (a form of time travel), then with the assigned user's name (e.g., "Alice") retrieved from a device inventory, and finally with Alice's role within the company (e.g., "manager in the engineering organization") and her real-time account status (e.g., active/suspended/deactivated) from an identity provider. This illustrated how disparate data sources could be seamlessly interwoven to provide a comprehensive human-centric view of an event.

Finally, the NXR Pattern demonstration focused on transforming a raw event into actionable "threat information." The example used was the securityd_authtrampoline process on macOS. The "after processing" data included a clear, human-readable description of the process ("an authentication prompt that is called by an API that says execute with privileges"), a mapping to relevant MITRE ATT&CK tactics and techniques, explicit references to external resources (e.g., "Objective-C blog"), and a dynamically calculated risk score. This visual proof highlighted how the data pipeline could generate rich, self-contained threat signals that serve as analytical signposts for security teams, making complex detections immediately understandable and actionable without requiring further SIEM-side enrichment.

These visual demonstrations served as compelling evidence of the practical benefits and transformative power of implementing these advanced ETL patterns directly within the data pipeline, making the enriched data readily available in the SIEM for immediate use.

Defensive Implications

▶ Watch: Technical Q&A on scalability (DynamoDB, serverless) and GenAI for schema norm... (27:00)

The "Reinventing ETL" approach proposed by Josh Liburdi carries significant defensive implications for security teams, fundamentally altering how they operate and respond to threats:

• Enhanced Alert Triage and Investigation Efficiency: By providing highly contextualized data directly in the SIEM, analysts no longer need to spend valuable time writing complex queries or manually correlating disparate data points. This drastically reduces the cognitive load, speeds up alert triage, and makes investigations more efficient and less prone to human error, especially during high-stress incidents.
• Mitigation of Data Decay: Performing real-time enrichment within the data pipeline ensures that contextual information (e.g., IP geolocation, user roles, device assignments, cloud metadata) is accurate at the moment of observation. This directly addresses the pervasive problem of data decay, leading to more reliable and trustworthy analysis.
• Democratization of Threat Intelligence and Knowledge: The NXR pattern, by transforming "threat information" into data, breaks down knowledge silos. Instead of threat signals being locked within specific teams or platforms, they become universally accessible and searchable within the SIEM. This fosters better collaboration between SOC, IR, and hunt teams, as they all operate from a shared, enriched dataset.
• Reduced Reliance on Deep Domain Expertise: With events enriched with human-readable descriptions, process lineages, and user context, analysts with varying levels of experience can more quickly understand the significance of an event. This lowers the barrier to entry for new team members and reduces the pressure on senior analysts to constantly interpret raw data.
• Flexible and Granular Risk Scoring: Moving risk scoring into the data pipeline allows for dynamic calculation of risk based on various entities—not just hosts and users, but also individual processes, internet servers, or even organizational teams. This provides a more nuanced understanding of risk, enabling defenders to prioritize threats more effectively.
• Proactive Threat Hunting and Research: Enriched data, especially with analytical signposts and references, provides a richer foundation for threat hunting. Hunters can easily pivot between different contextual elements (e.g., search for all events referencing a specific blog post or MITRE technique) to uncover subtle threats.
• Cost Optimization and Vendor Independence: Leveraging open-source solutions like Substation, particularly with serverless architectures, can significantly reduce the operational costs associated with data ingestion and enrichment. This also provides greater control over the data pipeline, reducing reliance on expensive commercial SIEM features for basic contextualization.
• Adoption of Modern Security Standards: The emphasis on unified data models and real-time contextualization pushes organizations towards adopting what Liburdi considers "table stakes" for modern detection and response, ensuring their security posture is aligned with best practices.
• Improved Decision Making: Ultimately, by providing security teams with high-quality, contextualized, and timely information, these ETL advancements lead to more informed and accurate decision-making during critical security operations.

Key Takeaways

• Current Security ETL is Broken: The traditional approach to security ETL is fundamentally flawed, leading to uncontextualized data, complex SIEM queries, analyst fatigue, and inaccurate conclusions.
• Data Decay is a Critical, Overlooked Problem: Data attributes (e.g., IP geolocation, user roles) become outdated over time, undermining the accuracy of security analysis. Real-time enrichment is essential to combat this.
• Real-time Contextualization in the Data Pipeline is Paramount: The most effective place to enrich data is within the data pipeline, seconds to minutes after observation, to ensure accuracy and provide immediate context before data reaches the SIEM.
• Unified Data Models are Non-Negotiable: Modern detection and response teams must adopt unified data models as a standard to ensure consistency, ease of analysis, and effective information sharing.
• Innovative ETL Patterns Transform Data: Patterns like "Time Travel" (self-enriching data for lineage), "Telephone" (cross-stream context sharing), and "NXR" (moving SIEM queries to the pipeline) dramatically enhance data with actionable intelligence.
• Open-Source Solutions Offer Powerful, Cost-Effective Alternatives: Projects like Substation demonstrate that advanced, real-time ETL capabilities can be built and deployed efficiently using open-source tools and serverless architectures, reducing reliance on expensive commercial vendors.

About the Speaker(s)

Josh Liburdi is a security professional known for delivering dense, information-rich talks that often include strong opinions and meta-commentary, typically without the aid of notes. He openly admits that while the topic of ETL might seem "boring" to many, he considers it profoundly important for the efficacy of security operations. Liburdi's professional experience includes working at Brex, where he has been instrumental in implementing the advanced ETL solutions discussed in his presentation, such as the open-source project Substation, which has been in production for over three years. His philosophy centers on making security "easy" and ensuring it is "effortless to derive insight from data," a goal he believes most security data vendors currently fail to achieve.