Flipping the Criticality Funnel, A Practical Path to Real Prioritization

Flipping the Criticality Funnel, A Practical Path to Real Prioritization

Speakers: Sophia Sanles-Luksetich (GitHub); Zachary Goldman (GitHub)

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=ZjKSqGkyZZI

Overview

In the fast-paced world of cybersecurity, effective vulnerability management hinges on accurate prioritization. However, as Sophia Sanles-Luksetich and Zachary Goldman from GitHub demonstrated at VulnCon, many organizations find themselves trapped in a "criticality funnel" where an overwhelming number of findings are labeled "critical," rendering true prioritization impossible. This talk, "Flipping the Criticality Funnel, A Practical Path to Real Prioritization," dives deep into GitHub's journey to overcome this pervasive challenge by developing a custom, transparent, and auditable internal risk scoring framework.

The core of their presentation outlined a meticulously designed approach that moves beyond generic CVSS scores or vendor-specific metrics. GitHub's solution focuses on building a unified risk score tailored to an organization's unique threat landscape, asset criticality, and compliance requirements. By sharing their practical methodology, including technical architecture, data integration strategies, and continuous tuning processes, Sanles-Luksetich and Goldman provided a blueprint for other security teams to reclaim control over their vulnerability backlogs, ensure actionable remediation, and foster trust across engineering, executive, and auditing stakeholders.

Background

▶ Watch: The problem: "Too many criticals, nothing is." (0:00)

The genesis of GitHub's initiative stemmed from a widespread industry problem: the "too many criticals" dilemma. As Sophia Sanles-Luksetich articulated, traditional vulnerability management often suffers from a fundamental flaw where various internal teams and external sources submit vulnerability data, each with their own definition of "high security risk." This leads to a chaotic environment where engineers are left to arbitrarily decide which "critical" to address first, ultimately paralyzing remediation efforts. GitHub's own data underscored this, revealing that a staggering 50% of their findings were classified as high-security risks, a clear indicator that their prioritization system was failing.

Zachary Goldman elaborated on the sheer scale of this challenge at GitHub. Their production database manages approximately a quarter of a billion findings over its lifetime, with an average of 200,000 new findings generated daily. This immense volume is compounded by data flowing in from 25 different sources and counting, each possessing its own unique enrichment pipeline, severity definitions, and data shape—ranging from standard CVSS scores to proprietary risk metrics and even simple prose descriptions. This disparity makes direct comparison and consistent prioritization virtually impossible. The speakers identified two key anti-patterns: the risk of offending teams by deprioritizing their "high" findings, and the misconception that reducing the number of criticals equates to "cooking the books" rather than making the system genuinely actionable. The underlying issue is that if a risk score isn't actionable, it effectively renders all findings, regardless of their nominal severity, as equally unaddressed.

Key Findings

▶ Watch: GitHub's scale: 250M findings, 200K daily. (4:00)

The central revelation from GitHub's experience is the successful "flipping" of the criticality funnel. By implementing their custom risk scoring framework, GitHub transformed its vulnerability landscape from one where high-security risks constituted 50% of findings to a state where criticals are now the least likely finding. This significant shift demonstrates that it is possible to achieve meaningful prioritization without compromising security, proving that making a risk score actionable is not about downplaying threats but accurately reflecting their true impact and urgency.

Another key finding is the efficacy of a unified risk score in normalizing diverse vulnerability data. GitHub's approach successfully harmonized inputs from over 25 different sources, integrating real-world threat intelligence such as EPSS (Exploit Prediction Scoring System) and CISA KEV (Known Exploited Vulnerabilities), while also factoring in granular asset criticality and exposure. This comprehensive aggregation allows for a holistic and context-aware assessment of risk. Crucially, the talk emphasized the importance of an open framework approach, which, while inspired by the structural skeleton of CVSS 3.1, is highly customizable. This adaptability allows organizations to tailor the risk score to their specific threat models, internal asset valuations, and compliance mandates, such as FedRAMP. Finally, the speakers underscored the non-negotiable value of transparency and auditability. By making the risk calculation process explicit, maintaining strict versioning, and providing detailed breakdowns of how scores are derived, GitHub fostered trust among engineers, executives, security teams, and auditors, ensuring that prioritization decisions are defensible and understood.

Technical Deep Dive

▶ Watch: Transparency, versioning, and auditability for risk scores. (6:00)

GitHub's solution centers on the principle of "one score to rule them all," a unified GitHub Internal Risk Score (GIHRS) designed to provide consistent, actionable prioritization across their vast and varied vulnerability landscape. The technical backbone of this system is a carefully constructed risk score calculator, divided into four main components to ensure transparency, auditability, and maintainability. An abstract base class enforces a consistent contract, guaranteeing uniformity across iterations. Each new iteration of the scoring logic receives its own version calculator file, meticulously tracking changes and maintaining a clear paper trail. A central risk map stores all weights and variables in an easily accessible location, facilitating diffs and understanding of scoring adjustments. Finally, risk score details are explicitly provided for every calculated score, showing each variable and how they were multiplied or added together, providing full transparency to engineers and stakeholders.

This calculator is strategically integrated into GitHub's vulnerability management pipeline. When finding data is ingested from various sources, it first flows through individual source workers to a central vulnerability aggregator. This aggregator groups related findings and generates a notification, typically in the form of a GitHub issue assigned to a specific team. The GIHRS calculation is hooked into this process before the notification is sent out. At this stage, all relevant metadata from related findings is combined, processed, and the custom risk score is computed and attached to the notification, dictating its priority and associated SLA.

Building this custom risk score is a three-step iterative process:

1. Taking Inventory: This initial phase involves identifying all existing data sources and their prominence, as this dictates the shape and availability of metadata. A thorough metadata check is then performed to understand what data is currently accessible and what additional data might be needed to enrich the score, particularly for compliance requirements like FedRAMP. The goal is to gather as much data as possible to create a complete risk picture.

1. Constructing the Risk Score: This is the core development phase, starting with a crucial consideration of the target audience:

• Engineers require transparency to avoid arguments and build confidence.
• Executives focus on ROI and backlog reduction.
• Security Teams are often eager to integrate their own metadata.
• Auditors demand auditability and clear justification for decisions.

The score is built upon metric groups, which are the highest-level categories always included in the calculation. GitHub uses three primary groups: Threat, Base Score, and Asset. The interaction between these groups is critical; GitHub opted for a simple multiplication (Threat Base Score Asset) to ensure that if any factor is zero, the overall risk score appropriately reflects a negligible risk. This also allows for interaction rules, such as Base Score and Asset working together to define impact. While the framework borrows the structural "skeleton" of CVSS 3.1 for its vector modeling, GitHub applies its own custom weights and values.

Key vectors (granular characteristics) include:

• Base Risk: Often derived from existing CVSS scores (e.g., a Python vulnerability with a CVSS 7.5). The talk noted challenges with CVSS 4.0 due to its survey-based nature making underlying characteristic modification difficult, favoring CVSS 3.1 for its flexibility. For non-CVSS sources, a simple conversion to low/medium/high/critical is often feasible, or a generic score bypass can be implemented if limitations exist.
• Threat: This is arguably the most dynamic and crucial vector. GitHub integrates:
• EPSS: Used as a quantitative measure, requiring internal thresholds (GitHub set a minimum of 0.001). EPSS scores are adjusted for the SLA duration by multiplying the score by itself for longer periods, and the data is refreshed frequently (every ~2 days).
• Qualitative Threat: A flexible "grab bag" for analyst input, policy changes, and specific weaponization data from vendors. This vector serves as a catch-all for external environmental factors or specific organizational concerns.
• Public Access/Credibly Exploitable Vulnerabilities: Treated as the highest threat level; beyond this point, a finding is considered an incident.
• Asset: Defines the value and sensitivity of the affected asset, often leveraging existing inventory attestation data. This includes:
• Confidentiality: PII, data sensitivity.
• Integrity: Considered particularly important due to risks like crypto mining or malware infrastructure, where even small assets can be leveraged for broader attacks.
• Availability: Often already tracked by internal systems.
• The framework is flexible enough to include other factors, such as physicality, if relevant to an organization.

To manage complexity, vectors are collapsed into higher-level representations:

• Base Risk remains CVSS 3.1.
• Threat is consolidated to the max threat observed across all sources to avoid double-counting overlapping data (e.g., EPSS and Tenable VPR both considering weaponization).
• Assets are treated as a cumulative sum of their requirements.

Weights are applied to these collapsed vectors, providing the final configuration for the score. These are considered the most open and frequently modified part of the process. For instance, GitHub weighted integrity higher in asset scoring due to its potential for widespread impact even on seemingly low-value assets. The result of this process is a transformation of a generic score (e.g., a 7.5 CVSS) into a context-specific, actionable score (e.g., a 3.0 low risk GIHRS).

1. Tuning: This is a continuous, iterative process that refines the risk score based on real-world outcomes. Key aspects of tuning include:

• Severity Bucketing: Aligning the custom risk score ranges with existing SLA (Service Level Agreement) definitions, as most teams base their remediation timelines on severity. Boundary points between severities are crucial areas of friction to monitor.
• Score Distribution Analysis: Examining the maximum possible score (often capped at 10 for UI purposes, even if the internal calculation goes higher), the median score (a gut check for overall reasonableness), and the mode (identifying the most common risk level).
• Example Scenarios: Using tools like LLMs (Large Language Models) to generate diverse scenarios and then "gaming them out" with the security team. This helps identify edge cases and tricky points where the score might not align with desired outcomes, prompting further adjustments to weights or vector definitions. An example given was a high base score with an exploit in the wild—does a 7.2 truly reflect the desired urgency?

The talk illustrated this integration with a screenshot directly from GitHub's platform, showing an engineer viewing a security finding with the GIHRS and its constituent vectors clearly listed, even if not all values were fully populated in the initial rollout phase. This visual demonstration underscores the practical application of their framework.

Demo / Proof of Concept

▶ Watch: Threat sources: EPSS, qualitative threat, and weaponization. (14:30)

While the talk did not feature a live, interactive demo in the traditional sense, the speakers provided a powerful visual demonstration of their system's practical application within GitHub's operational environment. Sophia Sanles-Luksetich presented a screenshot directly from the GitHub platform, showcasing how an engineer would interact with a security finding. This screenshot illustrated the integration of the GitHub Internal Risk Score (GIHRS), clearly listing all the contributing vectors and their current values for a specific vulnerability.

This visual evidence served as a compelling proof of concept, demonstrating that the theoretical framework had been successfully implemented and was actively providing actionable insights to engineers. The speakers noted that even in its initial rollout phase—focused primarily on correcting CV severity with threat data—the system had "increased our reliability of severity score incredibly." This practical, in-platform visualization validated the core premise of their talk: a custom, transparent risk score can significantly improve vulnerability prioritization and management.

Defensive Implications

▶ Watch: Continuous tuning, scenario planning, and LLM use. (22:00)

The GitHub team's approach to flipping the criticality funnel offers profound implications for defensive security postures. Firstly, it empowers security teams to achieve actionable prioritization, moving beyond the paralysis of "too many criticals" to focus resources on vulnerabilities that genuinely pose the highest risk to the organization. This precision allows for more effective resource allocation, ensuring that engineering teams are addressing the most impactful issues aligned with the company's specific risk tolerance and business objectives.

Secondly, the emphasis on transparency and auditability is a game-changer for fostering collaboration and trust. By openly sharing how risk scores are calculated, maintaining version control, and providing detailed breakdowns, security teams can significantly reduce friction with engineering counterparts, executives, and auditors. Engineers gain confidence in the assigned priorities, leading to less debate and faster remediation. Executives receive clear, data-driven insights into the true risk landscape and the ROI of security investments. Auditors, in turn, have a defensible and traceable record of prioritization decisions, which is critical for compliance frameworks like FedRAMP.

Furthermore, this framework provides organizations with strategic control over their vulnerability management destiny. Instead of passively accepting generic CVSS scores or vendor-assigned severities that may not reflect their unique environment, defenders can tailor their risk assessment to their specific crown jewels, threat intelligence feeds, and internal policies. This adaptability allows for continuous tuning in response to evolving threat landscapes, new compliance requirements, or shifts in organizational priorities, ensuring that the vulnerability management program remains agile and relevant. Finally, the process itself encourages a deeper dive into available metadata and data enrichment, leading to a more comprehensive understanding of assets, threats, and ultimately, a more robust and intelligent defensive strategy.

Key Takeaways

• Actionable Prioritization is Paramount: The pervasive problem of "too many criticals" renders vulnerability prioritization ineffective. The goal should be to create a risk score that is truly actionable, allowing teams to focus on the vulnerabilities that matter most to your organization.
• Build a Custom, Internal Risk Score: Relying solely on external or generic scores like raw CVSS is insufficient. Develop an internal, custom risk score that incorporates your organization's unique asset criticality, threat landscape, and compliance requirements (e.g., FedRAMP).
• Transparency and Auditability are Non-Negotiable: For stakeholder buy-in from engineers, executives, and auditors, the risk scoring process must be transparent, showing how scores are calculated. Maintaining strict versioning and a detailed paper trail is crucial for defensibility.
• Leverage and Customize Existing Frameworks: Use established models like CVSS 3.1 as a structural skeleton for defining metric groups and vectors, but apply your own specific weights and values. Integrate real-world threat intelligence, such as EPSS and CISA KEV, and factor in asset sensitivity.
• Collapse Overlapping Data for Accuracy: When integrating multiple threat intelligence sources, consolidate them (e.g., by taking the max threat) to avoid over-weighting and ensure the risk score accurately reflects the true overall threat, preventing double-counting of similar indicators.
• Risk Scoring is a Continuous Tuning Process: The definition and tuning of your risk score is not a one-time event. It requires continuous iteration, scenario testing (potentially using LLMs), and refinement to align with business objectives, evolving threats, and SLA requirements.

About the Speaker(s)

Sophia Sanles-Luksetich and Zachary Goldman are both from GitHub, bringing practical, enterprise-scale experience to the challenges of vulnerability management.

Sophia Sanles-Luksetich spearheaded the introduction of the problem statement, articulating the common pitfalls of traditional risk scoring and the psychological anti-patterns that hinder effective prioritization. She delved into the philosophical underpinnings of their custom risk score, emphasizing the critical importance of auditability, transparency, and tailoring solutions to an organization's specific needs. Her insights guided the audience through the detailed construction of the risk score, from defining metric groups and vectors to applying weights and considering various stakeholder perspectives.

Zachary Goldman provided the crucial context of GitHub's specific case study, illustrating the immense scale of their vulnerability data (hundreds of millions of findings, hundreds of thousands daily) and the complexity introduced by integrating data from over 25 diverse sources. He detailed the technical architecture of how their custom risk score is integrated into GitHub's vulnerability management pipeline, explaining the components of the risk score calculator and its seamless hook into their notification process. Together, their combined expertise offered a comprehensive and actionable blueprint for addressing one of the most pressing challenges in modern cybersecurity.