Death by Dashboards: Moving the Needle on What Actually Matters

Tim Medin

Wild West Hackin' Fest @ Mile High 2026 · Day 1

Overview

In his compelling talk, "Death by Dashboards: Moving the Needle on What Actually Matters," Tim Medin of Red Siege challenges the conventional wisdom surrounding cybersecurity metrics and Key Performance Indicators (KPIs). Medin argues that many organizations are trapped in a cycle of addressing highly visible, but ultimately low-impact, vulnerabilities while neglecting more significant, albeit less obvious, security risks. The core problem, he posits, lies in misaligned incentives, where security teams are often measured on activity rather than actual security posture improvement.

Watch on YouTube

Visual summary for Death by Dashboards: Moving the Needle on What Actually Matters by Tim Medin
Visual summary for Death by Dashboards: Moving the Needle on What Actually Matters by Tim Medin

Key moments

  1. 0:00 Introduction to the 'Death by Dashboards' problem
  2. 2:55 The 'visible vs. urgent' problem in security
  3. 4:30 Understanding the 'Cobra Effect' of misaligned incentives
  4. 10:05 Why using CVSS alone for priority is problematic
  5. 17:00 Actionable advice: Prioritize and break down remediation tasks
  6. 18:30 Using tradeoffs and gamification to improve security
  7. 23:35 Q&A: Policy is the root cause of dashboard problems

Death by Dashboards: Moving the Needle on What Actually Matters

Speakers: Tim Medin, Pen Tester, Red Siege

Conference: Wild West Hackin' Fest @ Mile High

YouTube: https://www.youtube.com/watch?v=WXkw1yC8YEQ

Overview

In his compelling talk, "Death by Dashboards: Moving the Needle on What Actually Matters," Tim Medin of Red Siege challenges the conventional wisdom surrounding cybersecurity metrics and Key Performance Indicators (KPIs). Medin argues that many organizations are trapped in a cycle of addressing highly visible, but ultimately low-impact, vulnerabilities while neglecting more significant, albeit less obvious, security risks. The core problem, he posits, lies in misaligned incentives, where security teams are often measured on activity rather than actual security posture improvement.

Medin, a seasoned penetration tester, draws upon decades of experience to highlight how current dashboard-driven approaches often lead to a "cobra effect," where efforts to improve security inadvertently make it worse or, at best, inefficient. He advocates for a fundamental shift in how security teams prioritize, measure, and communicate risk. This talk is crucial for blue teams, security managers, and anyone grappling with the challenge of demonstrating tangible security value amidst an overwhelming influx of vulnerability data.

The presentation provides a critical examination of widely adopted practices, particularly the reliance on CVSS (Common Vulnerability Scoring System) scores as a sole determinant of remediation priority. Medin urges a re-evaluation of security policies and metrics to focus on outcomes that genuinely enhance an organization's defensive capabilities against real-world threats, rather than merely clearing items from a dashboard.

Background

▶ Watch: Introduction to the 'Death by Dashboards' problem (0:00)

The modern cybersecurity landscape is characterized by an incessant stream of vulnerability reports, often aggregated into dashboards meant to provide a clear picture of an organization's security posture. However, as Tim Medin points out, these dashboards frequently create a false sense of progress, fostering a culture where "urgent" is confused with "visible." Issues that appear prominently on a dashboard, irrespective of their actual exploitability or impact, command immediate attention, while critical, systemic flaws that lack a numerical score or visual representation are often ignored.

Medin introduces the concept of the "cobra effect" to illustrate this phenomenon. Originating from a historical anecdote in British India, where a bounty on cobras led to people breeding them for profit, only to release them when the bounty was cancelled, the "cobra effect" describes how an incentive designed to solve a problem can inadvertently make it worse. In cybersecurity, this manifests when metrics, such as "mean time to fix," incentivize security teams to patch easy, low-impact vulnerabilities to meet targets, rather than tackling complex, high-impact issues. This leads to a superficial improvement in dashboard numbers but no genuine enhancement of security.

A key example of an "invisible" problem cited by Medin is the secure architecture of Active Directory. Misconfigurations or incorrect permissions within Active Directory are potent attack vectors for adversaries, yet they rarely trigger alerts or appear on standard vulnerability dashboards. Because these issues are not visible, they are often overlooked, allowing significant security gaps to persist. This highlights a fundamental disconnect between what is easily measurable and what is truly important for an organization's defense. The reliance on default vulnerability scanner outputs, without critical analysis, further exacerbates this problem, leading to a focus on theoretical risks over practical threats.

Key Findings

▶ Watch: Understanding the 'Cobra Effect' of misaligned incentives (4:30)

Tim Medin's central argument revolves around the critical disconnect between perceived security activity and actual security improvement. His key findings highlight how current practices, driven by flawed metrics and incentives, often lead organizations astray:

  1. The Illusion of Urgency: Dashboards prioritize "visible" issues, not necessarily "urgent" or "important" ones. This means security teams spend valuable time addressing vulnerabilities that show up in scanners like Nessus or Nexpose, even if they are theoretical or rarely exploited (e.g., TLS 1.1, specific PHP, Apache, Nginx, or IIS flaws), while neglecting deeper, systemic problems like Active Directory misconfigurations.
  2. CVSS Misapplication: The Common Vulnerability Scoring System (CVSS), while a useful input, is routinely misused as the sole determinant for remediation priority. Medin emphatically states that the creators of CVSS themselves advise against using it alone, emphasizing its focus solely on the vulnerability's attributes, not an organization's specific environment, compensating controls, or actual risk posture. This often leads to critical ratings for issues with no practical impact.
  3. Ineffective Incentives (The Cobra Effect): Metrics like "mean time to fix" incentivize speed over impact. Security teams will naturally prioritize the easiest vulnerabilities to fix, regardless of their actual importance, simply to meet their targets. This creates a "cobra effect" where the intended outcome (faster remediation of critical flaws) is undermined by the incentive structure.
  4. Wasted Resources: By focusing on high-CVSS, low-impact vulnerabilities, organizations expend significant resources (man-hours, patching cycles, change control) on activities that do not move the needle on actual security. This leads to burnout, frustration, and a failure to address the threats that truly matter.
  5. The Need for Recategorization: Medin stresses that organizations must develop the capability to recategorize and rescore vulnerabilities based on their unique context. This involves analyzing whether a specific function is actually in use, if compensating controls exist, and assessing the true exploitability and impact within their environment, moving beyond generic scanner outputs.

Technical Deep Dive

▶ Watch: Why using CVSS alone for priority is problematic (10:05)

Medin's critique of contemporary security practices delves into several technical areas, primarily focusing on the misapplication of vulnerability scoring and the resultant misdirection of technical effort. A cornerstone of his argument is the widespread misuse of the Common Vulnerability Scoring System (CVSS). He highlights that while CVSS provides a standardized way to rate vulnerability characteristics, it is explicitly not recommended by its creators as the sole basis for remediation priority. As Medin points out, the CVSS documentation itself states, "It is not recommended to use this alone to determine remediation priority" and "CVSS only takes into consideration the attributes of the vulnerability itself, doesn't include or your organizational stuff." Organizations frequently overlook this crucial caveat, creating policies that mandate remediation based purely on CVSS scores, often without considering environmental factors or compensating controls.

This leads to a situation where security teams are compelled to patch vulnerabilities that, in a practical context, pose minimal to no risk. Medin offers several vivid examples:

  • TLS 1.0 and 1.1: While these older cryptographic protocols are indeed weaker and flagged by compliance standards like PCI DSS as problematic, their practical exploitability for decryption in real-world scenarios is often theoretical. Medin challenges anyone to easily decrypt data transmitted over TLS 1.1, noting that "if you transmit 14 terabytes of data, I could decrypt two bytes," questioning the actual impact. Yet, these often receive high-severity ratings on dashboards.
  • PHP, Apache, Nginx, IIS Flaws: Medin, speaking from his experience as a pen tester for two decades, asserts that he rarely encounters or exploits these generic web server or scripting language flaws in real engagements. These often receive high CVSS scores because, theoretically, if a specific function is used with unsanitized user input (a rare, specific condition), it could be severe. However, without that specific context, they are often benign. Yet, they fill dashboards and drive patching efforts.

The problem is further compounded by the "mean time to fix" metric. While seemingly logical, it creates an incentive to tackle the easiest bugs, irrespective of their criticality. A security team measured by this metric will naturally prioritize a simple configuration fix over a complex architectural redesign, even if the latter addresses a far more significant risk. This leads to a scenario where "I pick the easiest bugs. I don't pick the difficult, hard one that might make a difference. I'm incentivized for speed."

Medin also touches upon the dynamic nature of risk, using the BlueKeep (CVE-2019-0708) RDP flaw as an illustration. Initially, BlueKeep was known to cause crashes but wasn't reliably exploitable. Years later, a reliable Metasploit module emerged. This evolution underscores that risk isn't static, but Medin cautions against paralyzing an organization with "what if someday this gets worse?" scenarios. Instead, he suggests tracking such issues with a "keep an eye on this" approach, rather than immediate, undifferentiated remediation.

The technical solution proposed by Medin is vulnerability recategorization and rescoring. This involves:

  1. Validation: Verifying if a vulnerability actually exists and is exploitable in the specific environment. For instance, for a PHP flaw, checking if the specific vulnerable function is actually in use.
  2. Contextual Risk Assessment: Evaluating the vulnerability against the organization's unique assets, data criticality, and existing compensating controls. This moves beyond a generic CVSS score to a tailored risk rating.
  3. Tiered Criticality: Categorizing systems into tiers (e.g., Tier 1, 2, 3) and applying different risk tolerances and remediation strategies. For example, some vulnerabilities might only be patched on Tier 1 systems, while lower tiers might be addressed through other means or accepted.

By implementing these steps, organizations can free up "patch monkeys" from "meaningless PHP patches" and redirect their efforts toward issues that genuinely "move a needle on a dashboard" in terms of keeping bad actors out. This is a call for a more intelligent, context-aware approach to vulnerability management, moving from a reactive, dashboard-driven process to a proactive, risk-informed strategy.

Demo / Proof of Concept

▶ Watch: Using tradeoffs and gamification to improve security (18:30)

The talk "Death by Dashboards" by Tim Medin does not include a live demonstration or a technical proof of concept. Instead, it focuses on conceptual arguments, real-world anecdotes from penetration testing experiences, and strategic advice for improving cybersecurity metrics and prioritization.

Defensive Implications

▶ Watch: Q&A: Policy is the root cause of dashboard problems (23:35)

Tim Medin's talk offers crucial insights for security defenders, urging a fundamental shift from reactive, dashboard-driven remediation to proactive, risk-informed strategies. The defensive implications are profound and actionable:

  1. Re-evaluate Vulnerability Prioritization: Defenders must move beyond solely relying on CVSS scores for remediation priority. As the CVSS creators themselves advise, these scores are merely an input. Organizations should develop their own internal risk scoring mechanisms that incorporate environmental factors, compensating controls, asset criticality, and actual exploitability. This means dedicating resources to recategorize and rescore vulnerabilities, verifying their existence and impact (e.g., checking if a specific PHP function is actually in use for a reported flaw).
  2. Focus on "Important" Over "Visible": Actively seek out and address security weaknesses that don't appear on standard dashboards. Active Directory misconfigurations, incorrect file share permissions, and insecure architectural designs are prime examples of "invisible" issues that attackers frequently exploit but often go unaddressed due to a lack of visibility.
  3. Implement Targeted Remediation: Instead of overwhelming business units with long lists of vulnerabilities, security teams should provide small, actionable lists (e.g., 3-5 high-impact items per month or quarter). This helps asset owners focus their efforts and increases the likelihood of successful remediation. For complex, foundational changes like Active Directory hardening, allocate it as a single, multi-month priority.
  4. Gamify Security Efforts: Introduce friendly competition among business units or teams by publicly tracking their progress on agreed-upon, high-impact security tasks. This can create a positive incentive, leveraging C-level engagement to drive remediation efforts.
  5. Strategic Trade-offs for Better Security: Implement policies that make it easier for users to adopt more secure practices. Medin highlights the example of password policies: instead of enforcing frequent rotation and complex character requirements (which NIST, Microsoft, and Google have shown to be counterproductive), offer users the trade-off of a much longer password (e.g., 14 characters) with no rotation. This simplifies the user experience while organically driving a more secure outcome.
  6. Redefine Key Performance Indicators (KPIs): Shift away from vanity metrics like "number of blocked attacks" or "blocked emails" (often referred to as "pew pew maps") which measure activity rather than impact. Instead, focus on meaningful indicators such as:
  • Mean Time to Respond (MTTR) to a breach or incident.
  • Early Kill Chain Ejection: How quickly an adversary can be detected and removed from the network at the earliest stages of an attack.
  • Employee Reporting Rates: Incentivize users to report suspicious activities, like phishing attempts, rather than just avoiding clicks.
  1. Address Policy as the Root Cause: Recognize that many flawed security practices are enshrined in organizational policies. Defenders must advocate for changing these policies at higher levels, even if it requires significant effort and C-level buy-in, to realign incentives and ensure security efforts are directed towards actual risk reduction.
  2. Continuous Validation of Metrics: Regularly review chosen metrics to ensure they are still driving the desired behavior and producing the intended outputs. Be vigilant for "cobra effects" where incentives might be inadvertently gamed.

By adopting these defensive strategies, organizations can move beyond the "death by dashboards" syndrome and cultivate a security program that truly enhances their resilience against evolving threats, optimizing resource allocation and fostering a culture of genuine security improvement.

Key Takeaways

  • Prioritize Real Risk Over Dashboard Visibility: Don't let scanner outputs or generic CVSS scores dictate your security priorities. Focus on vulnerabilities that pose actual, exploitable threats to your organization, even if they aren't highly visible.
  • Contextualize Vulnerability Scoring: Actively recategorize and rescore vulnerabilities based on your specific environment, existing compensating controls, and asset criticality. The CVSS is an input, not the sole determinant for remediation.
  • Beware the "Cobra Effect" of Metrics: Ensure your KPIs and incentives drive desired security outcomes, not just activity. Metrics like "mean time to fix" can lead to a focus on easy fixes rather than important ones.
  • Empower Teams with Manageable Tasks: Break down remediation efforts into small, actionable lists (e.g., 3-5 items) for business units or asset owners, making security improvements feel achievable rather than overwhelming.
  • Shift to Outcome-Based Metrics: Replace vanity metrics (e.g., blocked attacks) with measures of actual defensive capability, such as Mean Time to Respond (MTTR) to incidents or early adversary ejection from the kill chain.
  • Advocate for Policy Change: Recognize that deeply ingrained, ineffective security practices often stem from flawed organizational policies. Work to update these policies to align with modern, effective security principles.

About the Speaker(s)

Tim Medin is a seasoned penetration tester and the founder of Red Siege, a pen test company. With extensive experience in the field, Medin has taught widely and is known for his interactive speaking style. His expertise lies in understanding attacker methodologies and helping organizations improve their defensive posture by identifying and rectifying fundamental security weaknesses, often challenging conventional wisdom around security metrics and practices.

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

Medin is absolutely right, but he's preaching to the choir. Complaining that CVSS isn't a risk score and Nessus dashboards are useless is a conversation we've been having at the hotel bar for a decade. It's a competent sanity check for blue teams drowning in scanner vomit, but no new ground is broken here.

Heather Calloway (CISO) — MUST SEE

Medin delivers a necessary critique of vulnerability management programs that optimize for dashboard metrics over actual risk reduction. He rightly identifies that targets like Mean Time to Remediate (MTTR) incentivize teams to patch low-impact flaws to meet SLAs, leaving complex systemic risks untouched. This is required reading for leaders setting program KPIs.

→ Top-rated talks at Wild West Hackin' Fest @ Mile High 2026

All talks from Wild West Hackin' Fest @ Mile High 2026