What actually is Detection Engineering, and why do I need it?

What actually is Detection Engineering, and why do I need it?

Speakers: Regan Carey, Senior Security Engineer, Infotrust

Conference: CrikeyCon

YouTube: https://www.youtube.com/watch?v=z97ckdc5Ftg

Overview

In this insightful CrikeyCon presentation, Regan Carey, a Senior Security Engineer at Infotrust (formerly Interlock), dissects the often-misunderstood discipline of Detection Engineering (DE) within the context of a Security Operations Center (SOC). Drawing from six years of experience managing diverse Security Information and Event Management (SIEM) environments—including Splunk, CrowdStrike, Rapid7, and Microsoft platforms—Carey articulates the pervasive challenges SOCs face, such as alert fatigue, resource constraints, and the inherent complexity of SIEM operations. The talk serves as a foundational guide, moving beyond superficial definitions to establish DE as a critical, structured practice essential for modern security resilience.

Carey emphasizes that effective DE is not merely about deploying security rules; it's about systematically applying software development principles to the entire lifecycle of detection content. The presentation highlights the detrimental impact of "vibe-based detection engineering"—a reactive, ad-hoc approach prevalent in many organizations—and proposes a strategic shift towards a proactive, data-driven methodology. By integrating robust documentation, metrics, and a multidisciplinary perspective, DE transforms from a tactical function into a strategic pillar, connecting the SOC with broader business objectives and threat intelligence.

The core message resonates with any organization grappling with an overwhelming volume of alerts or an unclear understanding of its detection efficacy. Carey provides a clear roadmap, starting with fundamental prerequisites like comprehensive data source documentation and the establishment of meaningful SOC metrics. The talk culminates in an introduction to the Detection Engineering Behavior Maturity Model (DEBM), offering a tangible framework for organizations to assess their current DE capabilities and plot a course for continuous improvement, ultimately fostering a more resilient and efficient security posture.

Background

▶ Watch: Introduction and common SOC issues overview (0:00)

The landscape of modern cybersecurity operations is riddled with inherent complexities, particularly within the Security Operations Center (SOC). Regan Carey's talk underscores a critical and widespread issue: the struggle to effectively manage Security Information and Event Management (SIEM) systems. As organizations grow, SIEMs, whether on-premises solutions like Splunk or cloud-based platforms such as Microsoft Sentinel, become colossal, intricate beasts. Onboarding diverse log data from countless platforms, each with unique stakeholders and compliance requirements, quickly spirals into a balancing act of retention, cost, and security value. A common pitfall, as Carey points out, is the ingestion of vast quantities of data with little to no actual security utility, transforming the SIEM into an expensive "black box of configurations that nobody really remembers doing."

This operational quagmire is exacerbated by the prevalent "vibe-based detection engineering." This reactive approach manifests as ad-hoc rule modifications in response to false positives, tuning only when a rule "starts to spam," and an over-reliance on generic, vendor-provided detections that often generate thousands of irrelevant alerts daily. Such practices lead to rules running against non-existent or incorrectly parsed data, poor search performance, and detections targeting outdated Tactics, Techniques, and Procedures (TTPs) no longer relevant to the organization's threat landscape. Compounding these technical and procedural shortcomings is a fundamental lack of documentation for both incoming data sources and existing SIEM rules. Analysts are left to "vibe out" investigations, leading to inconsistent conclusions and, inevitably, the dreaded combination of alert fatigue and burnout within under-resourced SOC teams.

The problem, therefore, is multifaceted. It stems from the sheer complexity of SIEM management, the lack of a structured approach to detection development, and the human element of overburdened security teams. Without a deep understanding of the logged data, a clear mapping of data sources, and a systematic process for creating and maintaining detection rules, SOCs are condemned to a perpetual cycle of reactivity. This foundation of challenges sets the stage for the necessity of a disciplined approach like Detection Engineering, which seeks to inject order, predictability, and efficacy into an otherwise chaotic operational environment.

Key Findings

▶ Watch: Defining Detection Engineering in a SOC (10:00)

Regan Carey's talk delivers several pivotal findings that redefine the understanding and implementation of Detection Engineering (DE) within a Security Operations Center (SOC). Foremost among these is the assertion that DE is fundamentally about applying software development practices to the creation and management of detection rules. This analogy to a Software Development Lifecycle (SDLC), complete with requirements gathering, design, development, testing, and deployment phases, highlights DE as a structured, repeatable process rather than a series of ad-hoc actions.

A critical insight is that true DE is primarily a process issue, not merely a tooling problem. While tools like Sigma rules, Atomic Red Team, or Splunk Attack Range can be leveraged, they are effective only when integrated into a well-defined workflow. Carey emphasizes that significant maturity can be achieved through disciplined process adherence, even without complex automation initially. This process-centric view facilitates consistent documentation, change control, versioning, and rigorous testing before detections are deployed to the SIEM.

Another key finding is the concept of DE as a fusion center. This means DE must actively integrate diverse inputs to build robust, context-aware detections. These inputs include Cyber Threat Intelligence (CTI) (e.g., from Recorded Future or public research), internal threat modeling, feedback from red and purple team engagements, insights from incident response, and crucial operational metrics and feedback from SOC analysts. This holistic integration ensures that detections are tailored to an organization's specific threat landscape and operational realities, moving beyond generic vendor rules.

To measure and guide this journey, Carey introduces the Detection Engineering Behavior Maturity Model (DEBM), published by Elastic in November of the previous year. This framework provides quantitative and qualitative metrics to assess the maturity of a DE function, ranging from Tier 0 (foundational, or "vibe-based") to Tier 4 (highly sophisticated and automated). The DEBM's flexibility allows organizations to incrementally improve specific aspects of their DE program, even with budget constraints or limited appetite for large-scale investment, making it a practical guide for continuous improvement.

Finally, the talk highlights essential prerequisites for successful DE: comprehensive documentation of every data source flowing into the SIEM (what it is, how it's parsed, its purpose) and the diligent tracking of SOC metrics. Beyond simple true/false positive rates, organizations must understand Mean Time to Triage and Mean Time to Closure, and identify rules that trigger frequently without true positives, or conversely, those that rarely trigger but are critical. These foundational elements are crucial for building context-aware rules and data-driven tuning, shifting the SOC from a reactive stance to a proactive one.

Technical Deep Dive

▶ Watch: Detection Engineering as a 'Fusion Center' (13:00)

Regan Carey’s presentation meticulously unpacks the technical intricacies and operational deficiencies prevalent in many Security Operations Center (SOC) environments, laying the groundwork for understanding the necessity and mechanics of Detection Engineering (DE). At the heart of the problem is the sheer complexity of SIEM management. Whether dealing with traditional platforms like Splunk, next-gen solutions like CrowdStrike, or cloud-native offerings such as Microsoft Sentinel (which Carey specifically notes can be "even more complicated"), SOCs face immense challenges. These include the laborious process of onboarding diverse log data, balancing retention policies against escalating costs, and making critical decisions about what data holds genuine security value. Often, SIEMs become bloated with low-value data, leading to a "black box" of configurations that impedes effective security operations.

The prevailing "vibe-based detection engineering" is technically unsound and operationally inefficient. This approach is characterized by ad-hoc rule changes in response to false positives, reactive tuning, and an over-reliance on generic, vendor-provided detections. While vendors like Splunk and Elastic offer excellent starting points, enabling every default rule often results in an unmanageable deluge of alerts. Furthermore, rules frequently run against non-existent or improperly parsed data, leading to detections that never trigger or contribute to poor search performance. The use of outdated TTPs further degrades detection efficacy, as rules designed for past threats remain active against evolving adversaries.

To overcome these issues, true Detection Engineering advocates for the systematic application of software development practices. This involves a structured workflow akin to a CI/CD pipeline:

1. Requirements Gathering: Identifying specific threats, attack surfaces, and log sources.
2. Design Backlog: Prioritizing and planning detection development.
3. Development: Crafting detection rules, potentially using standardized formats like Sigma rules.
4. Testing: Rigorously validating rules using tools like Atomic Red Team or Splunk Attack Range to simulate adversarial behaviors.
5. Deployment: Implementing rules into the SIEM, ideally through automated processes (Detection as Code).
6. User Feedback Loop: Continuous tuning and refinement based on SOC analyst feedback and performance metrics.

Detection as Code is a powerful concept within DE, enabling version control, atomic configuration, and robust change management for detection rules. This ensures that every rule change is documented, justified, and tested, preventing the "black box" scenario and providing an auditable history of modifications.

A core technical element of DE is its function as a fusion center. This means integrating disparate data streams and intelligence sources to build truly context-aware detections. These inputs include:

• Cyber Threat Intelligence (CTI): Both commercial (e.g., Recorded Future) and open-source intelligence inform the understanding of current threats.
• Internal Threat Modeling: Understanding the organization's unique attack surface and critical assets.
• Red/Purple Team Engagements: Real-world adversarial simulations provide invaluable feedback on detection gaps.
• Incident Response (IR): Post-incident analysis identifies what detections failed or were missing.
• Business Stakeholders: Collaborating with line-of-business application owners ensures relevant logs are collected and specific attack surfaces are monitored.
• SOC Analyst Feedback: Operational insights on alert volume, false positives, and investigative challenges drive rule tuning.

Before embarking on advanced DE, foundational technical prerequisites are crucial. This includes comprehensive data source documentation: mapping every index, source type, and table within the SIEM to understand what data is coming in, from where, how it's parsed, and its security purpose. Tools like Detect for can assist in this mapping process. Additionally, robust SOC metrics are indispensable. Beyond simple true/false positive rates, organizations must track Mean Time to Triage (MTTT) and Mean Time to Closure (MTTC). These metrics help identify underperforming rules (e.g., those triggering hourly but never yielding a true positive over six months) and guide data-driven tuning decisions, moving away from reactive "vibe-based" adjustments.

The talk leverages the Detection Engineering Behavior Maturity Model (DEBM) by Elastic, which outlines a progression of technical sophistication:

• Tier 0 (Foundational): Characterized by ad-hoc, reactive rule deployment and minimal processes.
• Tier 1 (Basic): Involves having a defined rule set with documentation, all data sources recorded, and decent MITRE ATT&CK coverage (ensuring detections cover at least a tactic in the kill chain for each application/data source). Version control, even through a knowledge base, is present.
• Tier 2 (Gap Analysis): Focuses on deeper threat modeling and recording SIEM metrics (e.g., understanding if detections are skipping or dropping results), identifying specific gaps in coverage.
• Tier 3/4 (High End): Represents advanced capabilities, including regular threat emulation exercises (e.g., using Atomic Red Team), continuous red/purple teaming, deep CTI integration, and automated delivery of detections (requiring a full Detection as Code setup and leveraging projects like Sigma). Critically, Security Orchestration, Automation, and Response (SOAR) platforms are only truly effective at these higher tiers, requiring well-defined, reasonably tuned detection rules and pre-built response plans to function correctly. Implementing SOAR at lower maturity levels often leads to failure.

Carey underscores that while advanced tools and automation are appealing, the fundamental discipline of documentation and process adherence forms the bedrock of effective detection engineering. Ignoring these foundational elements in favor of "shiny tools" is a common trap that prevents true maturity.

Demo / Proof of Concept

▶ Watch: Advanced DEBMM tiers: Automation, CTI, and SOAR (19:00)

This particular talk focuses on conceptual frameworks, process definition, and strategic implementation rather than a live technical demonstration or proof of concept. Regan Carey's objective was to articulate what Detection Engineering truly entails and why it is a necessary discipline, providing a roadmap for organizations to mature their security operations. While the talk references various tools and techniques like Sigma rules, Atomic Red Team, and Splunk Attack Range as components of a mature DE program, these were discussed in the context of their theoretical application within the Detection Engineering Behavior Maturity Model tiers, not showcased in a practical demo. The emphasis is firmly on the underlying processes and methodologies required to build a robust detection capability.

Defensive Implications

▶ Watch: Detection Engineering is a process, not just tools (21:00)

The insights presented by Regan Carey offer profound implications for security defenders, providing a clear pathway to elevate Security Operations Center (SOC) capabilities beyond reactive fire-fighting. The primary defensive implication is the imperative to shift from a "vibe-based" and reactive approach to a proactive, structured, and data-driven Detection Engineering (DE) methodology. This transition is not merely an operational improvement but a fundamental enhancement of an organization's security posture and resilience.

First and foremost, defenders must prioritize comprehensive documentation. This includes meticulously mapping every data source entering the SIEM—identifying indexes, source types, tables, their origin, parsing methods, and most importantly, their security value. Equally crucial is documenting every detection rule, including its purpose, expected behavior, and a well-defined response plan. This foundational documentation combats knowledge silos, ensures consistency in investigations, and reduces the burden on analysts, particularly during critical 3 AM alerts. Without this, consistent and effective incident response is severely hampered.

Secondly, the adoption of a metrics-driven approach is non-negotiable. Defenders should move beyond basic true/false positive rates to track critical SOC metrics such as Mean Time to Triage (MTTT) and Mean Time to Closure (MTTC). By analyzing these metrics alongside SIEM-specific performance data (e.g., rules skipping results, search performance), organizations can identify inefficient or noisy detections. This data empowers defenders to proactively tune, disable, or enhance rules, thereby reducing alert fatigue and allowing analysts to focus on genuine threats. This also informs resource allocation and demonstrates the value of DE efforts to leadership.

Thirdly, integrating DE as a fusion center is critical. Defenders must actively seek input from diverse sources: Cyber Threat Intelligence (CTI), internal threat modeling, red and purple team engagements, Incident Response (IR) feedback, and insights from business unit stakeholders. This holistic integration ensures that detections are not generic but are tailored to the organization's specific threat landscape, critical assets, and attack surface. This allows for the development of context-aware detections that are significantly more effective against relevant threats.

Furthermore, defenders should leverage the Detection Engineering Behavior Maturity Model (DEBM) as a strategic tool. This framework provides a clear, actionable guide for assessing current DE capabilities (from Tier 0 to Tier 4) and setting realistic, measurable goals for improvement. It allows security teams to identify specific areas for development, whether it's enhancing MITRE ATT&CK coverage, implementing Detection as Code, or integrating automated testing with tools like Atomic Red Team. The DEBM also provides a common language to communicate progress and justify investment to leadership.

Finally, the talk offers a crucial caution regarding Security Orchestration, Automation, and Response (SOAR). While SOAR platforms promise significant efficiency gains, defenders must understand that their effective implementation is contingent upon a high level of DE maturity (Tier 3/4). Attempting to deploy SOAR without well-defined, tuned detection rules and pre-established response plans will inevitably lead to failure. Defenders should prioritize foundational DE processes and documentation before investing heavily in advanced automation, ensuring that the automation has a solid, reliable base to operate from. By embracing these defensive implications, organizations can build a more resilient, efficient, and proactive security operation.

Key Takeaways

• Detection Engineering (DE) is a Process, Not Just Tools: DE fundamentally involves applying software development principles to the creation, management, and refinement of detection rules, emphasizing structured workflows over ad-hoc, "vibe-based" approaches.
• Documentation is Foundational: Comprehensive documentation of all SIEM data sources (what they are, how they're parsed, their purpose) and every detection rule (including response plans) is a critical prerequisite for effective, consistent, and scalable security operations.
• Metrics Drive Maturity: Beyond basic true/false positives, tracking SOC metrics like Mean Time to Triage and Mean Time to Closure, alongside SIEM performance data, enables data-driven tuning and identifies ineffective or noisy rules, significantly reducing alert fatigue.
• DE as a Fusion Center: Effective DE integrates diverse inputs—Cyber Threat Intelligence (CTI), internal threat modeling, red/purple team feedback, incident response insights, and business stakeholder requirements—to build context-aware, tailored detections relevant to an organization's specific threat landscape.
• The DEBM Provides a Roadmap: The Detection Engineering Behavior Maturity Model (DEBM) offers a practical, tiered framework (from Tier 0/Foundational to Tier 4/Highly Sophisticated) for organizations to assess their current DE capabilities, set measurable goals, and strategically mature their detection program.
• Prioritize Process Over Shiny Tools: While advanced tools like Sigma, Atomic Red Team, and SOAR are valuable, significant DE maturity can be achieved by first establishing robust processes and documentation. Implementing advanced automation without these foundational elements is likely to be ineffective.

About the Speaker(s)

Regan Carey is a Senior Security Engineer at Infotrust, a company also known as Interlock prior to its rebranding. With six years of dedicated experience working within a Security Operations Center (SOC), Regan has developed deep expertise in managing complex Security Information and Event Management (SIEM) environments. His work primarily involves Microsoft security platforms, but his experience extends to a wide array of SIEM solutions including Splunk, CrowdStrike, and Rapid7. Throughout his career, Regan has been instrumental in implementing various SIEMs, migrating content between different platforms, and deploying numerous detection rules. This presentation at CrikeyCon marked Regan's first time speaking at a conference.