The Fault in Our Metrics. Rethinking How We Measure Detection & Response

Allyn Stott (Staff Engineer · Airbnb)

Swiss Cyber Storm 2024 · Day 1 · Arena

Overview

In his compelling Swiss Cyber Storm talk, "The Fault in Our Metrics. Rethinking How We Measure Detection & Response," Allyn Stott, a Staff Engineer at Airbnb, critically examines the pervasive issues with how security teams, particularly in Detection & Response (D&R), measure their performance. Stott argues that many traditional metrics are not only ineffective but actively harmful, leading to misguided decisions, wasted resources, and a failure to communicate true business value. He challenges the common practice of relying on easily quantifiable but ultimately meaningless data points, such as raw alert counts, which he dubs "operational burden" metrics.

Watch on YouTube · Slides

Visual summary for The Fault in Our Metrics. Rethinking How We Measure Detection & Response by Allyn Stott
Visual summary for The Fault in Our Metrics. Rethinking How We Measure Detection & Response by Allyn Stott

Key moments

  1. 0:00 Introduction and the 'metrics mess'
  2. 4:00 The 'operational burden' of alert counts
  3. 5:00 Introducing the SABER framework for better metrics
  4. 12:50 Pyramid structure for metrics at different altitudes
  5. 15:30 Introducing the TDR (Threat Detection and Response) Maturity Model
  6. 19:50 The 'cold plunge' analogy for changing metrics
  7. 23:00 Q&A: Reporting actionable data vs. raw data
  8. 27:00 Q&A: Shifting from 'storytelling' to real metrics

The Fault in Our Metrics. Rethinking How We Measure Detection & Response

Speakers: Allyn Stott, Staff Engineer, Airbnb

Conference: Swiss Cyber Storm

YouTube: https://www.youtube.com/watch?v=1t-uQnFkxz0

Overview

In his compelling Swiss Cyber Storm talk, "The Fault in Our Metrics. Rethinking How We Measure Detection & Response," Allyn Stott, a Staff Engineer at Airbnb, critically examines the pervasive issues with how security teams, particularly in Detection & Response (D&R), measure their performance. Stott argues that many traditional metrics are not only ineffective but actively harmful, leading to misguided decisions, wasted resources, and a failure to communicate true business value. He challenges the common practice of relying on easily quantifiable but ultimately meaningless data points, such as raw alert counts, which he dubs "operational burden" metrics.

Stott's presentation stems from his personal experience inheriting a "metrics mess" and realizing the profound impact that flawed measurement has on a security program's direction and funding. He highlights that while metrics are crucial for driving improvement, securing budget, and justifying headcount, the adage "what gets measured improves" carries an implicit warning: measuring the wrong things will lead to improvements in the wrong areas. The talk serves as a call to action for security professionals to abandon vanity metrics and adopt a more intentional, outcome-driven approach to measuring D&R effectiveness.

To address these challenges, Stott introduces two novel frameworks: the SABER framework for categorizing and aligning metrics with desired outcomes, and the Threat Detection and Response (TDR) Maturity Model for holistically assessing and advancing D&R capabilities. By presenting common mistakes and offering actionable alternatives, Stott empowers teams to move beyond superficial reporting and establish metrics that genuinely reflect their impact on organizational security, foster continuous improvement, and resonate with various stakeholders, from individual contributors to the board of directors.

Background

▶ Watch: Introduction and the 'metrics mess' (0:00)

Allyn Stott's journey into the intricacies of security metrics began, like many, with a mandate to report to leadership on security operations. His initial experience, described as inheriting a "metrics mess" where previous reports were "just made up," underscores a widespread problem in the industry. This lack of foundational rigor often leads security teams to scramble for easily available data, frequently resulting in the adoption of metrics that appear to show activity but lack strategic value. Stott references Carl Pearson's famous quote, "that which is measured improves," but adds a crucial caveat: what if you're measuring the wrong thing? Drawing from Hower and Kats' paper "Metrics: You Are What You Measure," he emphasizes that the metrics chosen will inevitably improve, and over time, an organization "will become what [it] measure[s]."

A primary culprit identified by Stott is the ubiquitous alert count. He labels this metric as "operational burden" or "we're doing things, it's crazy out there," noting its prevalence as the "heartbeat metric for security operations." While seemingly straightforward, alert count often provides a misleading picture. A decrease might indicate improved detection, reduced threats, or simply that logging has stopped. An increase could mean better detection, more attacks, or just more noise. Stott points out the common attempt to refine this by breaking it down into true and false positives, but questions its utility, asking, "I don't know what I was trying to say with this metric." Such metrics fail to provide actionable insights, making it difficult for teams to understand where to invest their efforts or how to demonstrate tangible improvements.

The core of this problem, Stott explains, lies in engineers' natural inclination to focus on technical tasks rather than strategic reporting, often viewing metrics as an "annoying PowerPoint I need to update every month." When security teams spend their days triaging alerts and responding to incidents, it's easy to lose sight of the overarching goals. This leads to a reduction of complex security work into simplistic, quantitative measures that don't capture the nuances of D&R, fail to connect to business outcomes, and ultimately hinder effective decision-making, budget allocation, and headcount justification. Stott's talk is a direct response to this pervasive issue, aiming to provide a structured path out of the "metrics mess" and towards more meaningful measurement.

Key Findings

▶ Watch: Introducing the SABER framework for better metrics (5:00)

Allyn Stott's talk identifies five critical "mistakes" that security teams commonly make when crafting and utilizing metrics, providing the foundation for his proposed solutions. These mistakes highlight the pitfalls of traditional measurement and underscore the need for a paradigm shift in D&R reporting.

  1. Losing Sight of the Goal: This is epitomized by the reliance on metrics like alert volume. Stott argues that simply tracking the number of alerts, even broken down by true/false positives, becomes a measure of "operational burden" rather than an indicator of security posture or improvement. It fails to convey what value is being delivered or where efforts should be focused. He emphasizes that metrics should clearly answer a question and tie back to a desired outcome.
  1. Using Quantities That Lack Controls: Many traditional metrics measure aspects of D&R that the team cannot directly influence or control, such as overall incident recovery time. Stott points out the high variability in such metrics, especially downstream in the response process, making them unhelpful for decision-making. He advocates for metrics that allow teams to filter out uncontrollable elements and focus on areas where they can make a direct impact.
  1. Thinking Proxy Metrics Are Bad: This mistake involves the pursuit of overly comprehensive and expensive metrics, such as attempting to measure detection coverage across the entire MITRE ATT&CK framework. Stott argues that while such an endeavor might seem ideal, it often consumes immense resources (years of effort, significant tooling investment) for a question that could be answered more efficiently. He suggests that focusing on prioritized, high-impact threats (proxy metrics) is a more pragmatic and effective approach.
  1. Not Adjusting to the Altitude: Stott highlights the common failure to tailor metrics to the audience. Presenting highly technical details like the MITRE ATT&CK framework to a board of directors, for example, is often ineffective. Metrics need to be contextualized for different levels of the organization, from individual contributors to executive leadership, connecting D&R activities to broader business impact.
  1. Asking "Why" Instead of "How": Stott observes that constantly asking "why" can lead to unproductive conversations and blame, particularly in incident reviews. Shifting to "how"—"how can we detect these threats sooner?" or "how can we respond faster?"—fosters a more positive, action-oriented, and solution-focused approach to problem-solving.

To counter these mistakes, Stott introduces two foundational frameworks:

  • SABER Framework: An acronym for Streamlining, Awareness, Vigilance, Exploration, and Readiness. This framework provides a structured approach to ensure metrics are tied to clear outcomes and measurable objectives, helping teams understand what they should be measuring and why.
  • Threat Detection and Response (TDR) Maturity Model: Building upon the Hunting Maturity Model (HMM) by David Bianco, this model offers a comprehensive way to assess and advance the maturity of a D&R program across key pillars like Observability, Proactive Threat Detection, and Rapid Response. It provides a common language and a roadmap for improvement, addressing the "how" of building a modern D&R program.

Together, these findings and frameworks offer a powerful methodology for security teams to move beyond flawed metrics and cultivate a measurement strategy that truly drives improvement and demonstrates value.

Technical Deep Dive

▶ Watch: Introducing the TDR (Threat Detection and Response) Maturity Model (15:30)

Allyn Stott's talk provides a robust framework for redefining security metrics, moving from arbitrary counts to actionable insights. At the heart of his proposed methodology are the SABER framework for metric categorization and the Threat Detection and Response (TDR) Maturity Model for program assessment.

The SABER Framework: Guiding Metric Creation

The SABER framework is designed to ensure that every metric answers a clear question and ties back to a specific outcome, preventing the creation of "operational burden" metrics. Each letter represents a category of outcomes:

  • S - Streamlining: Focuses on improving efficiency and accuracy through automation. Instead of just counting false positives, Stott advocates for measuring time spent on false positives. This shifts the incentive from merely tuning alerts to identifying where manual effort is highest and automating those processes. By tracking "human hours saved" through automation, teams can demonstrate tangible value. For example, if an alert takes 15 minutes of manual triage and is automated, that 15 minutes is a direct saving.
  • A - Awareness: Aims to raise awareness about learned threat intelligence, trends, and shared knowledge. Metrics here would involve tracking identified top threats, their prevalence, and how threat intelligence informs detection and response strategies. This moves beyond simply collecting threat feeds to demonstrating how that intelligence is actively shaping defensive posture.
  • V - Vigilance: Measures preparedness for top threats and the effectiveness of current detection capabilities. Instead of attempting a costly, comprehensive MITRE ATT&CK coverage map, Stott proposes a prioritized detection development approach. Teams should identify their top five most critical threats (e.g., based on external threat intelligence, incident trends, or the "what would make our Chief Privacy Officer cry the most" test). For each of these threats, they derive relevant MITRE ATT&CK techniques and sub-techniques. As detections are built and tested for these specific techniques, a prioritized ATT&CK coverage map organically develops, far more cost-effectively than a full-scale effort.
  • E - Exploration: Tracks insights gained from proactive threat hunts. This category focuses on what new gaps or vulnerabilities are discovered during exploratory activities. Metrics could include the number of new gaps identified, the severity of findings, or how threat hunts inform new detection development.
  • R - Readiness: Assesses the speed and completeness of incident response capabilities. Stott criticizes generic Mean Time To Respond (MTTR) metrics due to their high variability and lack of control. Instead, he suggests breaking down response time across all phases of an incident (e.g., detection, containment, eradication, recovery). Crucially, he introduces the "make it zero" principle: for each phase and incident type, define an expected, acceptable duration. Any time spent within that expectation is "zero" in the metric, highlighting only the deviations or overruns. This allows teams to focus on specific bottlenecks they can control and improve, filtering out built-in, unavoidable time.

The Threat Detection and Response (TDR) Maturity Model

To provide a holistic view of a D&R program's capabilities and guide its evolution, Stott developed the TDR Maturity Model. This model expands upon David Bianco's influential Hunting Maturity Model (HMM), extending its scope beyond just hunting to encompass the full spectrum of D&R.

The TDR Maturity Model is structured around three core pillars:

  1. Observability: Focuses on having the necessary visibility and context within the environment. This includes capabilities related to logging, data collection, and data quality.
  2. Proactive Threat Detection: Encompasses activities like threat intelligence collection, detection engineering, and threat hunting. This pillar ensures the program is actively seeking and identifying threats.
  3. Rapid Response: Deals with the ability to quickly and effectively respond to incidents, including playbook development, automation, and incident management.

Within these three pillars, the model defines 14 specific capabilities. For each of these capabilities (e.g., "detection engine capability," "playbook completeness"), teams score four different areas:

  • Process: Are there defined, optimal processes in place for this capability?
  • Tools: Are the necessary tools available and effectively utilized?
  • Documentation: Is the capability well-documented, clear, and accessible?
  • Testing: How is the effectiveness and logic of this capability validated?

Each of these four areas is rated on a scale from Initial to Leading, providing a granular assessment. For instance, for "detection engine capability," an "Initial" score might mean ad-hoc detection creation, while "Leading" implies a well-defined process, robust tooling, comprehensive documentation, and automated testing for detection logic.

The TDR Maturity Model allows teams to visualize their current state ("where are we now?") and define a target state ("where do we want to be by next year?"), based on planned projects. This visualization, often presented as a radar or spider chart, provides a clear, high-level overview of maturity across the pillars, while retaining the underlying detail for each capability. It helps connect the work being done to tangible improvements in program maturity and provides a common language for discussing progress.

Metric Altitude and Evolution

Stott emphasizes the importance of adjusting to the altitude of the audience. He proposes a Metric Pyramid:

  • Top (Business Level): Focus on Northstar metrics that reflect business impact, such as "reducing the cost of an incident" or "increasing the cost for an attacker to cause impact."
  • Middle (Coverage & Effectiveness): Metrics here relate to how well security controls and detections cover threats and how effective they are.
  • Bottom (Tool Performance): Detailed metrics on the performance and utilization of specific security tools.

This pyramid ensures that leadership receives high-level, business-relevant information, while practitioners have the granular data they need to operate and improve. Furthermore, Stott advocates for metric expiration. As D&R programs mature and automate more tasks, some traditional metrics (like raw alert counts) become obsolete and should be retired to make way for more relevant measures, reflecting the continuous evolution of security operations.

Demo / Proof of Concept

▶ Watch: The 'cold plunge' analogy for changing metrics (19:50)

While Allyn Stott's talk did not feature a live technical demonstration in the traditional sense, he effectively presented conceptual "proofs of concept" through visualizations of his proposed frameworks. These served as clear examples of how teams could implement and benefit from his methodologies.

Specifically, Stott demonstrated how the Threat Detection and Response (TDR) Maturity Model could be visualized. He showed a radar or spider chart illustrating a program's current maturity ratings across the three pillars (Observability, Proactive Threat Detection, Rapid Response) and their underlying 14 capabilities. Crucially, this visualization also included a projection of where the team planned to be by the end of the year, based on their strategic projects. This graphical representation clearly communicates progress and impact to various stakeholders, showing if the work being done is actually moving the needle on maturity.

Similarly, though not a live demo, Stott's detailed explanation of the SABER framework and the "make it zero" principle served as a conceptual walkthrough. He described how a graph of "time spent on false positives" could track human hours saved by automation, or how breaking down incident response times by phase, filtering out expected durations, could pinpoint specific bottlenecks. These explanations, supported by illustrative graphs in his slides, provided a tangible understanding of how to apply his principles to real-world D&R data.

The speaker also mentioned providing a link tree at the end of the talk, which includes access to the full TDR maturity model and the presentation slides. This allows the audience to independently explore and apply these frameworks, serving as a practical extension of the talk's conceptual demonstrations.

Defensive Implications

▶ Watch: Q&A: Shifting from 'storytelling' to real metrics (27:00)

Allyn Stott's insights offer profound implications for security defenders across all levels, from individual analysts to C-suite executives. By rethinking how D&R effectiveness is measured, organizations can make more informed decisions, optimize resource allocation, and genuinely improve their security posture.

For SOC Analysts and Detection Engineers:

  • Focus on Actionable Metrics: Abandon vanity metrics like raw alert counts. Instead, track metrics that directly reflect human effort saved by automation (e.g., time spent on false positives), or the efficiency of specific detection rules. This incentivizes building quality detections and robust automation rather than merely processing a high volume of alerts.
  • Utilize "Make It Zero": When analyzing incident response times, apply the "make it zero" principle. Break down response into distinct phases and establish expected durations for each. By filtering out acceptable time, analysts can immediately identify and prioritize bottlenecks or deviations from expected performance, leading to targeted improvements in playbooks or tooling.
  • Prioritize Detection Development: Instead of aiming for exhaustive but costly MITRE ATT&CK coverage, focus on the organization's top 5 threats. This pragmatic approach ensures that detection engineering efforts are aligned with the most significant risks, providing a higher return on investment and a more defensible posture against actual threats.

For Security Managers and Directors:

  • Adopt the SABER Framework: Use SABER (Streamlining, Awareness, Vigilance, Exploration, Readiness) to categorize and align D&R metrics with strategic objectives. This helps in communicating the "why" behind each metric and demonstrating how D&R activities contribute to broader security goals and business outcomes.
  • Leverage the TDR Maturity Model: Implement the Threat Detection and Response (TDR) Maturity Model to assess the program's current state and define a clear roadmap for improvement. This model provides a common language for discussing capabilities across Observability, Proactive Threat Detection, and Rapid Response, broken down by Process, Tools, Documentation, and Testing. It's an invaluable tool for justifying budget, headcount, and project prioritization, as it clearly articulates areas of strength and weakness.
  • Adjust to Metric Altitude: Ensure metrics presented to leadership are tailored to their perspective. Use the Metric Pyramid to translate technical D&R work into business impact (e.g., cost reduction, attacker deterrence) for the board, while retaining granular technical metrics for operational teams.
  • Collaborate Cross-Functionally: Recognize that D&R metrics can highlight issues beyond the immediate team. If a repeating incident leads to long recovery times, the best solution might involve working with other security teams (e.g., Enterprise Security) to implement new preventative controls, rather than solely trying to improve D&R response.

For All Security Professionals:

  • Embrace Change ("Cold Plunge"): Be prepared to challenge and retire old, misleading metrics. The initial shock of changing established reporting practices will give way to greater clarity and more effective program operation.
  • Shift from "Why" to "How": When encountering problems or incident after-action reviews, reframe questions from "Why did this happen?" to "How can we prevent this from happening again?" or "How can we detect/respond faster?" This fosters a proactive, solution-oriented mindset.
  • Continuous Evaluation: Regularly review and "expire" metrics that no longer serve a useful purpose or have been rendered obsolete by automation or process improvements. Metrics should evolve with the program.

By adopting Stott's principles, defenders can transform their approach to D&R measurement, creating a system that not only reflects their true value but also actively guides them towards a more mature, efficient, and resilient security posture.

Key Takeaways

  • Metrics Drive Improvement, So Measure Wisely: The fundamental principle is that what gets measured improves, but critically, security teams must ensure they are measuring the right things. Focusing on vanity metrics or those that reflect "operational burden" (like raw alert counts) will lead to improvements in areas that don't genuinely enhance security or business value.
  • Focus on Actionable and Controllable Metrics: Prioritize metrics that provide clear answers to specific questions and relate to aspects the team can directly influence. Principles like "make it zero" for incident response times help filter out uncontrollable variables, allowing teams to identify and address specific bottlenecks.
  • Prioritize and Be Pragmatic Over Comprehensive: Instead of attempting to achieve exhaustive (and expensive) coverage of frameworks like the entire MITRE ATT&CK matrix, focus on the organization's top 5 most critical threats. This targeted approach allows for more efficient resource allocation and a stronger defense against the most relevant risks.
  • Tailor Metrics to Your Audience (Altitude): Different stakeholders require different levels of detail. Use a "Metric Pyramid" to present high-level business impact to leadership (e.g., reduced cost of incidents) while maintaining granular technical metrics for operational teams. This ensures effective communication and alignment across the organization.
  • Leverage Frameworks for Maturity and Alignment: Utilize structured frameworks like the SABER framework to categorize and align metrics with desired outcomes (Streamlining, Awareness, Vigilance, Exploration, Readiness). Implement the Threat Detection and Response (TDR) Maturity Model to holistically assess program capabilities across Observability, Proactive Threat Detection, and Rapid Response, providing a roadmap for growth and clear communication of progress.
  • Embrace Change and Continuous Evolution: Be prepared to challenge and retire old, ineffective metrics. The initial discomfort of "taking the cold plunge" to change reporting practices will ultimately lead to greater clarity and more impactful measurement. Metrics should evolve as the security program matures and automates.

About the Speaker(s)

Allyn Stott is a Staff Engineer at Airbnb, specializing in enterprise security, threat detection, and incident response. Based in Austin, Texas, he brings a wealth of practical experience from the front lines of security operations. Stott's professional philosophy, as highlighted in his talk, emphasizes learning from mistakes and continuously seeking better ways to approach complex security challenges. He is dedicated to improving the way security teams measure their effectiveness, advocating for metrics that drive genuine improvement and clearly communicate value.

Reviews

Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT

Metrics talks are usually where security programs go to die, but Stott actually delivers a pragmatic framework for D&R. He murders vanity metrics like raw alert counts and MTTR, replacing them with measurements that might actually survive contact with a real incident.

Heather Calloway (CISO) — STRONG ACCEPT

Every CISO inherits a metrics mess at some point. Stott provides a highly pragmatic, unsentimental framework for stripping out vanity SOC data and replacing it with numbers that actually justify headcount and tool spend. Mandatory viewing for your director of security operations before the next quarterly business review.

→ Top-rated talks at Swiss Cyber Storm 2024

All talks from Swiss Cyber Storm 2024