Measurement matters

Measurement matters

Speakers: Michael Daniel, President and CEO, Cyber Threat Alliance

Conference: Virus Bulletin

YouTube: https://www.youtube.com/watch?v=3T-TPV1iKzM

Overview

In his compelling Virus Bulletin presentation, "Measurement matters," Michael Daniel, President and CEO of the Cyber Threat Alliance, tackles a profound and persistent challenge in cybersecurity: the inability to effectively measure the state of security beyond individual organizational boundaries. Daniel argues that while conventional wisdom suggests that the cyber landscape is perpetually worsening, there is a fundamental lack of quantifiable data to substantiate this claim or, more critically, to understand where improvements might be occurring or where resources should be directed. The talk champions the development of a concise, relevant, and manageable set of metrics that can provide actionable insights for national policymakers, industry leaders, and even boardrooms.

Daniel, drawing on his extensive experience, including as a former White House presidential adviser, highlights the urgent need for a framework to assess cybersecurity posture at regional, national, and ecosystem levels. He posits that without robust measurement, policy decisions are made in the dark, and investments lack clear justification. The presentation outlines a practical approach to identifying proxy metrics aligned with established cybersecurity frameworks like the NIST Cyber Security Framework, aiming to shift the discourse from anecdotal observations to data-driven analysis.

The significance of this work cannot be overstated. In an era where cyber threats are increasingly sophisticated and their economic impact is undeniable, understanding the collective effectiveness of defensive measures is paramount. Daniel's initiative, a collaborative effort with various global organizations, seeks to provide the tools necessary to gauge progress, identify systemic weaknesses, and ultimately foster a more resilient global cyber ecosystem. By proposing concrete, albeit proxy, measures, the talk lays the groundwork for a future where cybersecurity strategy is informed by empirical evidence rather than mere speculation.

Background

▶ Watch: Challenge: Measuring cybersecurity beyond organizational level (0:00)

The pervasive sentiment that "today is worse than yesterday and tomorrow will be worse than today" dominates discussions around cybersecurity. However, as Michael Daniel points out, this widely held belief lacks concrete, overarching metrics to support it. While individual organizations might track their own security posture, there remains a significant void in understanding the collective state of cybersecurity at regional, national, or even global ecosystem levels. This absence of objective measurement hinders effective policy-making, strategic investment, and the ability to discern the true impact of defensive efforts. Daniel recounts facing this very question from a US president: "How do we actually know if it's actually getting worse?" The inability to provide a data-driven answer underscores the criticality of this problem.

Developing a meaningful set of cybersecurity metrics for such broad scopes presents several formidable challenges. Daniel identifies four key hurdles:

1. Coverage: Given the vastness of cybersecurity, how much of its "cybersecurity-ness" can be adequately represented by a limited number of metrics (ideally 8-10, fitting on a briefing slide)?
2. Relevance: Are the chosen metrics truly relevant to the decision-makers who will use them? Do they inform decisions at the policy or board level?
3. Availability: Many aspects of cybersecurity are difficult to measure due to a lack of available data. This forces reliance on existing data, but as Daniel illustrates with the analogy of a "drunk looking for their car keys under the street lamp because that's where the light is but their car keys are somewhere else," using readily available but irrelevant data can be misleading.
4. Manageability: Can the data be continuously collected, absorbed, and managed over time to generate consistent trends?

To address these challenges, Daniel has collaborated with several prominent groups, including the World Economic Forum's Global Future Council on Cyber Security, Professor Jason Healey and his students at Columbia University, Cyber Green, Zurich Reinsurance, and the Aspen cyber groups. This extensive collaboration has led to pressure-testing various proposed metrics and frameworks, with some results appearing in forthcoming publications.

The foundational approach to selecting metrics involves using an established framework. While Daniel acknowledges that the specific framework might be less important than simply having one, he primarily utilizes the NIST Cyber Security Framework (CSF) due to his familiarity with it. Other frameworks, such as a simpler "threat, vulnerability, and impact" model (as used by Jay Healey), are also valid. The NIST CSF's functions—Identify, Protect, Detect, Respond, Recover, and Govern—provide a structured way to categorize and assess different aspects of cybersecurity, offering a logical scaffolding upon which to build a national or ecosystem-level measurement system.

Key Findings

▶ Watch: "Protect" metrics: Exploited vulnerabilities older than a year (5:50)

The central finding of Michael Daniel's talk is that despite the inherent difficulties, it is indeed possible to develop a coherent and actionable set of proxy metrics to measure the state of cybersecurity at national and ecosystem levels. These metrics, while not always perfectly precise, can collectively provide crucial trend data, enabling policymakers and industry leaders to make more informed decisions. The talk proposes specific, data-driven indicators aligned with the NIST Cyber Security Framework functions, moving beyond anecdotal evidence to a more analytical understanding of cybersecurity posture.

Daniel's work demonstrates that:

1. A limited, relevant set of metrics is achievable: By focusing on key indicators that reflect underlying security behaviors and outcomes, it's possible to distill the complexity of cybersecurity into a manageable number of data points suitable for high-level decision-making.
2. Existing frameworks can guide metric development: The NIST CSF, or similar risk management frameworks, provides a robust structure for categorizing and understanding different facets of cybersecurity, making it easier to identify relevant measures for each area.
3. Proxy measures can be highly informative: Even when direct measurement is impossible, well-chosen proxy metrics can effectively indicate progress or degradation in cybersecurity. For example, the percentage of exploited vulnerabilities older than a year serves as a proxy for patching effectiveness and attacker effort.
4. Trend data is more critical than point-in-time precision: The true value of these metrics lies in observing their movement over time, allowing for the assessment of policy impacts and overall trajectory of cybersecurity resilience. This enables a pre/post analysis of policy implementations, such as cyber insurance mandates or software liability laws, to determine their actual effect.
5. Economic impact quantification is a powerful motivator: Measuring the collective economic loss from cyber incidents provides a compelling argument for increased investment, mirroring the success of the financial services industry in justifying security spending.

Essentially, the key finding is a robust conceptual framework and a set of proposed metrics that, when consistently applied and tracked, can transform the understanding of cybersecurity from a qualitative, often pessimistic, narrative into a quantitative, evidence-based discipline.

Technical Deep Dive

▶ Watch: "Detect" metrics: Incident rates and mean time to detect (7:50)

Michael Daniel's proposed measurement framework systematically maps potential proxy metrics to the functions of the NIST Cyber Security Framework (CSF): Protect, Detect, Respond, Recover, and Govern. He notes a particular challenge in identifying a robust metric for the "Identify" function that has survived pressure testing, indicating an ongoing area of research. The objective is not necessarily precise numbers, but rather the ability to track trends over time and assess the impact of policy decisions.

Protect Function

The "Protect" function focuses on developing and implementing appropriate safeguards to ensure the delivery of critical infrastructure services. Daniel proposes two key metrics here:

1. Percent of exploited vulnerabilities older than a year: This metric aims to quantify the prevalence of long-standing, unpatched vulnerabilities that are still actively being exploited.

• Rationale: A high percentage indicates poor patching hygiene across the ecosystem, allowing attackers to leverage well-known, older exploits without expending resources on developing new ones or zero-days. As this percentage decreases, it suggests that defenders are more effectively patching, forcing attackers to invest in newer, more sophisticated exploits. This effectively raises the cost of attack for adversaries. Daniel emphasizes that this number should ideally be going down.
• Example (conceptual): If 40% of all exploited vulnerabilities in a given year are CVEs older than 12 months, this suggests a significant defensive weakness. Reducing this to 10% would indicate a substantial improvement in collective patching efforts.

1. Distribution of exploited vulnerabilities: This metric examines whether exploited vulnerabilities are clustered in specific areas (e.g., memory-related issues) or are more evenly distributed across different types (hardware, software, configuration, etc.).

• Rationale: A healthy, resilient ecosystem would ideally not have a disproportionate concentration of weaknesses in one area. If, for instance, a "ridiculous percentage" of exploited vulnerabilities are memory-related (as Daniel notes is currently the case), it points to a systemic weakness. A more even distribution would suggest that defensive efforts are diversified and addressing a broader spectrum of potential attack vectors, making it harder for attackers to find a single, pervasive point of failure.
• Example (conceptual): Instead of 70% of exploits targeting memory corruption, a more even distribution might see 20% memory, 20% web application, 20% network misconfigurations, 20% authentication bypass, etc., indicating a more robust and diverse defensive posture.

Detect Function

The "Detect" function focuses on developing and implementing activities to identify the occurrence of a cybersecurity event.

1. Rate of significant cyber incidents across your economy/nation: This metric aims to track the frequency of major cyber incidents impacting a defined scope.

• Rationale: While defining "significant" is a challenge, many countries are establishing mandatory reporting regimes (e.g., the US's CERSIA – Cyber Incident Reporting for Critical Infrastructure Act, India's mandatory reporting law, and the EU's NIS-2 directive). These initiatives will provide standardized data, allowing for consistent measurement over time. A declining rate of significant incidents would indicate improved collective detection and prevention.
• Challenges: Harmonizing definitions of "significant" across different jurisdictions remains a hurdle, but the trend towards mandatory reporting is a positive step.

1. Mean Time to Detect (MTTD): This measures the average duration it takes for organizations to identify an attacker's presence within their networks.

• Rationale: Often referred to as part of "dwell time," a lower MTTD is directly correlated with reduced impact and attacker success. Daniel recalls average dwell times around 300 days in the past, highlighting that such extended periods grant adversaries immense opportunities for lateral movement, data exfiltration, and system compromise. Driving this number down is a critical indicator of improved detection capabilities and overall security maturity.

Respond Function

The "Respond" function focuses on developing and implementing appropriate activities to take action regarding a detected cybersecurity incident.

1. Mean Time to Respond (MTTR): This metric measures the average time from detection of an incident to its full containment and eradication.

• Rationale: Complementing MTTD, MTTR assesses the efficiency of incident response processes. A shorter MTTR means less time for attackers to cause damage after being detected. This metric is crucial for minimizing the operational and economic impact of incidents. Reducing MTTR signifies more mature and effective incident response capabilities across the ecosystem.

Recover Function

The "Recover" function focuses on developing and implementing appropriate activities to maintain plans for resilience and to restore any capabilities or services that were impaired due to a cybersecurity incident.

1. Economic loss from cyber incidents: This is a comprehensive measure of the collective financial impact of cyber incidents on an economy.

• Rationale: This metric goes beyond direct costs like ransom payments to include lost productivity, lost profits, reputational damage, and the valuation of stolen intellectual property. Daniel argues that while precise numbers might be elusive, the rough order of magnitude is what truly matters for national policymakers. He draws a parallel to the financial services industry, which has historically excelled at cybersecurity because it could clearly quantify the "I can invest $5 million in a cyber tool, or I can lose $100 million in fraud" equation. Applying this logic at a national level provides a powerful justification for cybersecurity investment and helps assess whether current spending is adequate.

Govern Function

The "Govern" function, a relatively newer addition to the NIST CSF, focuses on establishing and monitoring the organization’s cybersecurity strategy, policies, and processes.

1. Number of unfilled cybersecurity positions: This serves as a proxy for the overall ability to manage and govern cybersecurity across an ecosystem.

• Rationale: A growing number of unfilled positions indicates a widening skills gap, suggesting a worsening capacity to address cybersecurity challenges. Conversely, a shrinking gap, whether through increased hiring, improved training, or leveraging emerging technologies like AI, would signify improved governance and a strengthening cybersecurity posture. This metric reflects the human capital necessary to implement and sustain security measures.

Identify Function

Daniel explicitly states that the "Identify" function, which focuses on developing an organizational understanding to manage cybersecurity risk to systems, assets, data, and capabilities, has proven challenging for developing a robust, pressure-tested metric. While acknowledging its importance, he highlights that a suitable ecosystem-level proxy measure has not yet been established in their collaborative work.

Overall Approach and Impact

The power of these proposed metrics lies in their ability to be tracked over time, generating trend data rather than static snapshots. This enables the crucial analysis of pre- and post-policy implementation, allowing governments and industries to empirically assess whether specific mandates (e.g., mandatory cyber insurance) or voluntary efforts (e.g., industry-led security initiatives) are actually having their intended effect on the measured security posture. This data-driven approach moves cybersecurity from a realm of qualitative conjecture to quantitative analysis, providing a foundation for evidence-based policy-making and strategic resource allocation.

Demo / Proof of Concept

▶ Watch: "Govern" metric: Unfilled cybersecurity positions and "Identify" gap (11:00)

While Michael Daniel's talk did not present a live demonstration of a fully implemented, real-time national cybersecurity measurement system, it did provide conceptual visualizations of how the proposed metrics could be presented. Daniel explicitly mentioned that the numbers shown in his example tables and charts were "totally made up" for illustrative purposes, designed to give an idea of what the data might look like.

The presentation included simplistic graphical representations, such as bar charts and trend lines, to convey how metrics like the "percent of exploited vulnerabilities older than a year" or the "distribution of exploited vulnerabilities" could be displayed visually. These conceptual graphics served as a proof-of-concept for the presentation layer of such a system, emphasizing the importance of conveying complex information quickly and effectively to high-level decision-makers. The goal was to show that even with a handful of well-chosen metrics, it is possible to create clear, digestible dashboards that highlight trends and areas of concern, fulfilling the requirement that such reports "fit on a page" or a single briefing slide. The focus was on the potential for visual communication of these measurements rather than a functional demonstration of data collection or processing.

Defensive Implications

▶ Watch: Using metrics to measure cyber policy impact (13:00)

Michael Daniel's proposed cybersecurity measurement framework carries significant implications for defenders across various organizational and governmental levels. By offering a structured approach to quantifying security posture, the framework empowers both individual entities and national bodies to make more informed, data-driven decisions.

For individual organizations and their security teams, these metrics provide a powerful lens for internal assessment and communication with executive leadership and boards.

• Prioritize Patching: The "percent of exploited vulnerabilities older than a year" metric directly incentivizes robust patch management programs. Defenders should prioritize timely application of security updates, especially for known exploited vulnerabilities. This metric highlights that neglecting older vulnerabilities is a low-cost win for adversaries.
• Diversify Defenses: The "distribution of exploited vulnerabilities" encourages a holistic approach to security architecture. Organizations should analyze their own vulnerability landscape to avoid over-reliance on defenses against a single class of vulnerability (e.g., memory exploits). This implies investing in diverse security controls across hardware, software, network, and human elements.
• Enhance Detection Capabilities: The focus on "Mean Time to Detect (MTTD)" underscores the critical need for advanced threat detection systems, robust security monitoring, and skilled analysts. Defenders should continuously refine their ability to identify intrusions quickly, reducing attacker dwell time and minimizing potential damage.
• Strengthen Incident Response: The "Mean Time to Respond (MTTR)" metric emphasizes the importance of well-defined, practiced, and efficient incident response plans. Organizations must invest in automation, clear communication protocols, and regular incident response drills to ensure swift containment and recovery.
• Quantify Economic Impact: The "economic loss from cyber incidents" metric encourages organizations to develop methodologies for quantifying the financial impact of security breaches. This data is invaluable for justifying cybersecurity investments, demonstrating return on investment (ROI) to leadership, and securing necessary resources.

For national and regional policymakers and governments, the framework offers tools for strategic planning, resource allocation, and policy evaluation.

• Evidence-Based Policy Making: Governments can use these metrics to assess the effectiveness of national cybersecurity strategies, mandates (e.g., cyber insurance requirements), and voluntary industry initiatives. This allows for a shift from reactive policy-making to proactive, evidence-based adjustments.
• Justify National Investment: Quantifying the collective "economic loss from cyber incidents" provides a compelling argument for national-level investments in cybersecurity infrastructure, research and development, and workforce development. It helps allocate budgets more effectively by demonstrating the tangible costs of inaction.
• Address Workforce Gaps: The "number of unfilled cybersecurity positions" metric highlights a critical national security and economic vulnerability. Governments and educational institutions should collaborate to develop comprehensive strategies for training, recruiting, and retaining cybersecurity talent, including exploring innovative solutions like AI integration.
• Promote Incident Reporting: The reliance on standardized incident reporting (e.g., CERSIA, NIS-2) necessitates government leadership in establishing clear reporting frameworks, ensuring data quality, and fostering trust with reporting entities. This collective data is essential for generating accurate national-level metrics.
• Foster Cross-Sector Collaboration: The ecosystem-level nature of these metrics implies that no single entity can improve the overall posture alone. Governments should facilitate information sharing, threat intelligence exchange, and collaborative defensive efforts across critical infrastructure sectors and industries.

In essence, Daniel's work provides a blueprint for defenders to move beyond a qualitative understanding of risk to a quantitative, outcome-oriented approach. By focusing on measurable indicators, defenders can better communicate value, prioritize efforts, and ultimately build more resilient and defensible cyber environments.

Key Takeaways

• The need for objective measurement is critical: Cybersecurity discourse often relies on anecdotal evidence; robust, ecosystem-level metrics are essential to understand the true state of security and inform policy.
• A concise set of relevant metrics is achievable: Despite the complexity, it's possible to identify a manageable number of proxy measures that provide meaningful insights for national policymakers and board-level decision-makers.
• Established frameworks provide structure: The NIST Cyber Security Framework (CSF) offers a valuable organizational structure for categorizing and developing these metrics across functions like Protect, Detect, Respond, Recover, and Govern.
• Specific proxy metrics can guide action: Proposed metrics like the "percent of exploited vulnerabilities older than a year," "Mean Time to Detect/Respond," "economic loss from cyber incidents," and the "number of unfilled cybersecurity positions" offer actionable insights into collective security posture.
• Trends and policy impact are paramount: Tracking these metrics over time to identify trends and assess the pre/post impact of policy decisions is more valuable than any single precise number at a given point in time.
• Economic quantification drives investment: Articulating the collective economic loss due to cyber incidents is a powerful motivator for national investment and improved security practices, mirroring success in the financial sector.

About the Speaker(s)

Michael Daniel is the President and Chief Executive Officer of the Cyber Threat Alliance (CTA), a non-profit organization focused on improving the cybersecurity of the global digital ecosystem through trusted, automated, and timely information sharing. Prior to his role at CTA, Daniel served as a White House Presidential Adviser, where he held the position of Cybersecurity Coordinator on the National Security Council. In this capacity, he was responsible for developing and implementing national cybersecurity policy and strategy for the United States. His extensive background in the US government, particularly in national security and cybersecurity, has provided him with a deep understanding of the challenges associated with measuring and improving cybersecurity at national and ecosystem levels, which forms the core of his "Measurement matters" presentation.