Schrödinger’s Detection: Finding the "Zombie" Rules in Your SIEM

Schrödinger’s Detection: Finding the "Zombie" Rules in Your SIEM

Speakers: Gowthamaraj, Detection Engineer, MetaB

Conference: fwd:cloudsec

YouTube: https://www.youtube.com/watch?v=tWy7Ceh1c98

Overview

In the dynamic landscape of cybersecurity, Security Information and Event Management (SIEM) systems are the bedrock of detection and response. However, the efficacy of these systems hinges entirely on the quality and accuracy of their detection rules. Gowthamaraj, a Detection Engineer at MetaB, sheds light on a pervasive and often silently crippling issue: "zombie rules." These are detection rules that appear healthy and deployed within a SIEM dashboard—showing "all green"—but fundamentally fail to trigger when actual malicious activity occurs. His fwd:cloudsec talk, "Schrödinger’s Detection: Finding the 'Zombie' Rules in Your SIEM," introduces a critical framework and a tool, Sigma Lens, designed to proactively identify and mitigate these insidious threats to an organization's defensive posture.

Gowthamaraj's research underscores a significant industry challenge, citing alarming statistics: CardinalOps reported in 2024 that 18% of SIEM rules are broken, a figure projected to rise to 28% broken or unused by 2025. Furthermore, an attack simulation by AttackIQ revealed that only one in seven detection rules actually trigger during real-world scenarios. These figures highlight a glaring gap between perceived detection coverage and actual defensive capability. This talk is crucial for blue teams, detection engineers, and security leaders who rely on SIEMs, providing actionable insights and tools to ensure their detection investments are truly effective against evolving threats.

Background

▶ Watch: Defining "Schrödinger's Detection" and "zombie rules" (0:00)

The concept of a "zombie rule" is central to Gowthamaraj's presentation. Unlike rules that explicitly error out, a zombie rule is functionally dead, providing a false sense of security. It might be deployed, show as active in the SIEM, and even pass superficial checks, but it fails to identify the specific malicious behaviors it was designed to detect. This silent failure can be far more dangerous than an outright error, as it leaves an organization vulnerable without their knowledge.

Gowthamaraj categorizes zombie rules into three primary types:

• Schema Zombies: These rules suffer from issues related to event fields or the underlying log schema. This might occur if a field specified in the rule does not exist in the ingested logs, is misspelled, or has changed due to an API update. The rule silently fails to match any events because the expected data structure is absent or incorrect.
• Logical Zombies: This category refers to rules where the detection logic itself is flawed. For example, a rule might attempt to match contradictory conditions (e.g., create user and delete user in the same event where they cannot coexist), or the combination of conditions is simply incorrect for the intended threat. These rules might technically execute but will never find a valid match.
• Coverage Zombies: Perhaps the most insidious, these rules provide a false sense of comprehensive coverage. A rule might claim to detect "exfiltration from AWS S3 buckets" but only cover a fraction of the possible scenarios or specific API calls, leaving significant gaps. This leads to a dangerous overestimation of an organization's defensive capabilities.

To illustrate a schema zombie, Gowthamaraj uses an example of an ECS credential endpoint rule. This rule aims to detect when an ECS task accesses credential endpoints. Its logic checks for events like register task definition, describe task definition, or run task, along with a field container definition commands. The critical flaw is that container definition commands only exists in the register task definition event. Therefore, while the rule might trigger for register task definition events, it will fail to trigger for run task events, which represent a highly probable attack vector, because the required field is missing from those logs. This silent failure demonstrates a dangerous gap in coverage despite the rule appearing active.

To address these issues, Gowthamaraj proposes a detection testing pyramid, inspired by the traditional testing pyramid but tailored for security detections. At the apex are red team and purple team activities, which are highly effective but resource-intensive and often have long backlogs, making them unsuitable for testing every rule. Below this is Breach and Attack Simulation (BAS), which involves executing attack commands in a controlled environment. While better, building and maintaining such environments for every possible attack is still a significant undertaking. Further down are end-to-end (E2E) and unit testing, where sample events are fed into detections. These are easier but still reactive. The foundational layer, and the focus of his work, is static schema validation. This proactive approach ensures that the fundamental elements of a detection rule—like field names and data types—are valid before deployment or before an API change renders them obsolete. Without this foundational layer, API changes (e.g., in AWS logs) can silently break detections, leaving organizations vulnerable for extended periods, especially for rules written years ago.

Key Findings

▶ Watch: Classifying zombie rules: schema, logical, and coverage (3:17)

Gowthamaraj's core finding is the pervasive and under-addressed problem of "zombie rules" within SIEM environments, posing a significant risk to an organization's security posture. His research, culminating in the development of Sigma Lens, demonstrates that a proactive, foundational approach to detection quality is not just beneficial but essential.

The talk highlights several critical findings:

• Prevalence of Zombie Rules: Industry data, reinforced by Gowthamaraj's own findings, indicates that a substantial percentage of SIEM rules are non-functional or provide inadequate coverage. The statistics of 18% broken rules (CardinalOps 2024), 28% broken or unused rules (CardinalOps 2025 projection), and only 1 in 7 rules triggering during simulations (AttackIQ) underscore the scale of the problem.
• Classification of Zombie Types: The categorization into schema zombies, logical zombies, and coverage zombies provides a structured way to understand and address different failure modes, moving beyond a generic "broken rule" label.
• Necessity of Static Schema Validation: The detection testing pyramid emphasizes that relying solely on higher-level testing (red teaming, BAS, E2E) is insufficient. The foundational layer of static schema validation is crucial for catching errors early and preventing silent failures caused by external factors like API changes.
• Effectiveness of Sigma Lens: The development of Sigma Lens as a static analysis tool proves that automated, shift-left validation is feasible and highly effective. When run against 3,000 public Sigma rules, Sigma Lens identified 8 verified zombie rules, demonstrating its capability to find real-world flaws even in widely reviewed repositories. This leads to the prediction that 20-30% of rules in production SIEMs could be zombies.
• Comprehensive Rule Quality Scoring: Sigma Lens introduces a scoring mechanism for detection rules, evaluating them across multiple dimensions: schema (30%), logic (25%), evasion (20%), and false positive (15%). This allows for a granular assessment of rule quality beyond simple pass/fail, enabling prioritization of remediation efforts.
• Role of LLMs in Future Analysis: The talk identifies the potential for Large Language Models (LLMs) to enhance detection analysis, particularly in identifying sophisticated evasion techniques and potential false positives, by providing environmental context and reasoning capabilities.

These findings collectively present a compelling case for integrating automated static analysis into the detection engineering lifecycle, shifting the focus from merely writing more rules to ensuring the quality and resilience of existing and new detections.

Technical Deep Dive

▶ Watch: Practical example: ECS credential endpoint zombie rule (4:10)

Gowthamaraj's solution, Sigma Lens, is an architectural framework and a tool designed to perform static schema validation and comprehensive analysis of detection rules. The primary goal is to identify zombie rules by scrutinizing their underlying structure, logic, and potential for evasion or false positives, proactively.

The architecture of Sigma Lens is straightforward yet powerful:

1. Sigma Rule Ingestion: The process begins with ingesting a Sigma rule. Sigma rules are a generic and open signature format that allows security analysts to describe detection methods in a vendor-agnostic way. A Sigma rule specifies the log source and event types it targets (e.g., AWS CloudTrail).
2. Provider Registry Mapping: Sigma Lens maps the ingested Sigma rule to a provider registry. This registry contains information about specific log sources and services (e.g., AWS as the provider, and CloudTrail as the service).
3. Schema Database (Schema DB): Once the provider and service are identified, Sigma Lens queries a Schema DB. This database stores the expected schema for various log sources and services. For instance, for AWS CloudTrail events, it would contain all the possible fields, their data types, and their relationships for specific API calls (e.g., CreateAccessKey or RunTask). The power of this DB is amplified by community contributions; if an API changes, a community member can submit a Pull Request (PR) to update the schema, ensuring the database remains current.
4. Analysis Engine: With the rule and its corresponding schema, the analysis engine performs several checks:

• Field Presence: It verifies if all fields specified in the Sigma rule's detection section actually exist in the schema for the targeted log source and event types. This helps identify schema zombies where fields might be misspelled or deprecated.
• Logic Validation: It checks the logical consistency of the rule. While not a full semantic check, it can flag obvious contradictions or inefficient logic that would prevent a rule from ever firing. This targets logical zombies.
• Metadata Quality: It assesses the completeness and accuracy of rule metadata (e.g., mapping to MITRE ATT&CK TTPs, product information). High-quality metadata is increasingly crucial for AI-driven security operations to prevent "hallucination" and provide accurate context.
• Evasion Analysis (Heuristic & LLM-driven): Initially, this is heuristic-based, looking for common evasion patterns. In future iterations (post 5/16), LLMs will be integrated. By feeding the LLM context about the environment and the rule, it can suggest alternative ways an attacker might achieve the same objective that the current rule doesn't cover, thus identifying coverage zombies. For example, creating an AWS admin user might be possible through multiple API calls, and the LLM could highlight if the rule only covers one.
• False Positive Analysis (Heuristic & LLM-driven): Similar to evasion analysis, heuristics identify common false positive patterns. LLMs can then be used to analyze the rule in the context of typical organizational behavior to predict potential legitimate activities that might trigger the rule, allowing for "shift-left" false positive mitigation.

Workflow Example (AWS CloudTrail Rule):

Consider a Sigma rule designed to detect CreateAccessKey events in AWS CloudTrail.

1. Step 1: Sigma Lens identifies the provider as AWS and the log source as CloudTrail.
2. Step 2: It then identifies the specific service and event, in this case, IAM and CreateAccessKey.
3. Step 3: From the Schema DB, it retrieves all fields associated with IAM CreateAccessKey events.
4. Step 4: The analysis engine checks if all fields referenced in the rule's detection section (e.g., eventName, userIdentity.type, responseElements.accessKey.status) are present in the retrieved schema. If a field is missing or misspelled (e.g., responseElements.accessKey.staus), it's flagged as a schema zombie.
5. Step 5: Logical checks ensure the conditions are valid.
6. Step 6: Finally, an LLM might be invoked (with environment context) to suggest if CreateAccessKey can be evaded or if its detection could lead to false positives based on specific IAM policies or automation.

The scoring system assigns weights to different aspects of rule quality: schema (30%), logic (25%), evasion (20%), and false positive (15%). This allows for a composite quality score, enabling detection engineers to prioritize rules for improvement. The LLM-driven components, while promising, are noted to be highly dependent on the quality of prompts and the environmental context provided.

Demo / Proof of Concept

▶ Watch: Introducing the detection testing pyramid for rule quality (5:50)

Gowthamaraj demonstrated the practical application and efficacy of Sigma Lens by running it against a substantial dataset of 3,000 public Sigma rules. The testing process, which initially took several hours due to API limits, was refined to produce a comprehensive dashboard of rule quality.

The results from this large-scale analysis were presented, showcasing the average quality scores:

• Heuristic Analysis: Achieved an average quality score of 82%. This score reflects the checks performed on schema, logic, and metadata without the advanced reasoning capabilities of LLMs.
• LLM-Enhanced Analysis: When LLMs were incorporated for evasion and false positive analysis, the average quality score dropped to 72%. Gowthamaraj advised taking the LLM-enhanced score "with a pinch of salt," acknowledging that evasion and false positive detection are highly environment-dependent, and the public Sigma rules may not have been optimized for specific environmental contexts. Nevertheless, this reduction indicates the LLM's ability to uncover more nuanced or context-specific flaws.

Specific examples were highlighted to illustrate Sigma Lens's capabilities:

• AWS RDS Rule (Schema Zombie): The tool successfully identified a non-existent field within an AWS RDS rule. The rule was referencing a field that simply did not exist in the actual AWS logs for the intended events. Sigma Lens not only flagged this as a critical issue but also suggested the correct field to use, effectively turning a dead rule into a functional one.
• AWS Similar RDS Rule (Schema Zombie): Another AWS RDS rule exhibited a similar problem, where an incorrect field was mapped. Sigma Lens accurately pinpointed the non-existent field and recommended the appropriate alternative, demonstrating consistency in identifying schema-related issues.
• Critical Non-existent Field Finding: In one particularly surprising instance, Sigma Lens identified a field in a Sigma rule that was entirely non-existent in AWS. Gowthamaraj expressed satisfaction that the tool was performing as intended, catching even critical, fundamental errors that could completely nullify a detection's effectiveness.

The fact that Sigma Lens could identify 8 verified zombie rules within widely used and community-contributed public Sigma repositories is a powerful testament to its utility. If such fundamental errors exist in highly scrutinized public rules, it strongly suggests that an even higher percentage—Gowthamaraj estimates 20% to 30%—of proprietary rules deployed in private production SIEMs are likely to be zombies, quietly failing to provide protection. The demo effectively showcased Sigma Lens's ability to bring these hidden failures to light, prompting organizations to address them proactively rather than discovering them during a real incident.

Defensive Implications

▶ Watch: Introducing Sigma Lens for static schema validation (8:10)

The insights and tools presented by Gowthamaraj have profound implications for defensive security operations, particularly for blue teams and detection engineers. The central message is a call to action to shift left in detection quality assurance, moving away from a reactive "hope for the best" approach to a proactive, systematic one.

Here are the key defensive implications:

1. Prioritize Foundational Quality: Defenders must recognize that the most critical layer of detection quality is the foundational static schema validation. Waiting for red team or purple team exercises to find broken rules is too late and too resource-intensive. Implementing checks at the earliest stages of the detection lifecycle, ideally in CI/CD pipelines, ensures that rules are sound before deployment.
2. Integrate Sigma Lens (or Similar Tools) into CI/CD: Organizations should integrate tools like Sigma Lens into their Continuous Integration/Continuous Deployment pipelines. This allows for automated validation of every new or modified detection rule against known schemas. If a rule contains a non-existent field, logical error, or metadata deficiency, the CI/CD pipeline should fail, preventing the deployment of a zombie rule.
3. Proactive Schema Management: Establish a robust process for managing and updating schemas for all ingested log sources. This includes actively monitoring API changes from cloud providers (e.g., AWS, Azure, GCP) or internal systems. While providers may not explicitly notify about field changes, a community-driven schema database (like the one envisioned for Sigma Lens) or internal automation can help keep schemas current. LLMs can play a role here by monitoring documentation for changes and suggesting updates.
4. Shift Focus from Quantity to Quality: The industry often emphasizes the number of rules or the breadth of coverage. However, Gowthamaraj's talk highlights that a large quantity of rules, if many are zombies, provides a false sense of security. Defenders should shift their focus to ensuring the quality and efficacy of existing detections, ensuring they actually trigger when intended.
5. Regular Audits of Existing Rules: Beyond CI/CD for new rules, organizations should periodically run static analysis tools against their entire existing rule base (e.g., weekly or monthly). This helps identify detection drift—rules that become zombies over time due to external changes (API updates, log format changes) or internal refactoring.
6. Enhance Rule Metadata: With the increasing adoption of AI in security operations, high-quality and comprehensive metadata (e.g., accurate MITRE ATT&CK mapping, clear descriptions, known false positives/evasions) is paramount. Sigma Lens's ability to assess metadata quality helps ensure that AI tools receive reliable input, preventing "garbage in, garbage out" scenarios.
7. Leverage LLMs for Deeper Analysis: While currently in development for Sigma Lens, the future integration of LLMs for evasion and false positive analysis offers a powerful defensive capability. By feeding LLMs context about an organization's specific environment, they can help identify sophisticated attack paths not covered by existing rules or flag legitimate activities that might trigger alerts, allowing defenders to refine rules preemptively.

By adopting these defensive strategies, organizations can significantly improve the reliability of their SIEM detections, reduce the operational burden of false positives, and ultimately enhance their ability to detect and respond to real threats more effectively.

Key Takeaways

• Zombie rules are a widespread and silent threat: A significant percentage of detection rules in SIEMs are functionally dead, providing a false sense of security. Industry statistics suggest 18-28% are broken or unused, and only 1 in 7 trigger during real attacks.
• Foundational static analysis is critical: Organizations must prioritize static schema validation at the earliest stages of the detection lifecycle, as advocated by the detection testing pyramid, to prevent silent failures caused by API changes or incorrect field mappings.
• Sigma Lens offers a proactive solution: Gowthamaraj's tool, Sigma Lens, provides a framework for automated, comprehensive analysis of detection rules, identifying schema, logical, and coverage zombies through heuristic and (future) LLM-driven checks.
• Integrate detection quality into CI/CD: Implement automated checks for detection rule quality within CI/CD pipelines to ensure that only validated, high-quality rules are deployed, preventing the introduction of new zombies.
• Manage and maintain log schemas: Proactively store, manage, and update log source schemas (e.g., for AWS CloudTrail, S3) to ensure detection rules remain aligned with the actual data being ingested, preventing detection drift over time.
• Shift from rule quantity to rule efficacy: Focus on the quality and functional accuracy of detection rules rather than merely the volume. Regularly audit existing rules to identify and remediate those that have become zombies.

About the Speaker(s)

Gowthamaraj is a Detection Engineer at MetaB. He has been actively researching the problem of "zombie rules" in SIEMs and has developed Sigma Lens, a tool aimed at addressing this challenge. His work presented at fwd:cloudsec reflects his dedication to improving the efficacy and reliability of security detections, leveraging his expertise in detection and response to build practical solutions for the cybersecurity community.