Disparate Data, Distorted Decisions: Vendor Data Bias in CTI

Disparate Data, Distorted Decisions: Vendor Data Bias in CTI

Speakers: Eli Woodward, Threat Intel Advisor, Team Cymru

Conference: FIRSTCON

YouTube: https://www.youtube.com/watch?v=Fu2ibIYAvAc

Overview

In this compelling talk at FIRSTCON, Eli Woodward, a Threat Intel Advisor at Team Cymru, exposed a critical, yet often overlooked, challenge in the cybersecurity industry: vendor data bias in Cyber Threat Intelligence (CTI). Drawing from years of experience in incident response (IR) and law enforcement, Woodward meticulously deconstructed the annual threat reports from leading security vendors, revealing significant discrepancies in their identification of top initial access vectors. The core argument is that while these reports are valuable, their data sources, collection methodologies, and internal taxonomies are inherently biased, leading to a distorted view of the threat landscape and potentially misinformed security investments by organizations.

The genesis of this research stemmed from a personal observation: conflicting vendor reports being used to justify tool purchases, highlighting a "vendor shopping" mentality. Woodward's subsequent deep dive into how major IR firms—such as CrowdStrike, Mandiant, Unit 42 (Palo Alto), and Verizon—source, aggregate, and report their data uncovered systemic issues. These issues, ranging from client demographics and insurance panel influence to non-disclosure agreements (NDAs) and varying telemetry, create a fragmented and often contradictory picture of global cyber threats.

This talk is crucial for anyone involved in CTI, security leadership, or strategic decision-making within an organization. It urges practitioners to move beyond simply consuming vendor reports at face value and instead adopt a critical, analytical approach. By understanding the underlying biases and limitations, organizations can triangulate information more effectively, align threat intelligence with their specific risk profiles, and make more defensible and impactful security investments. Woodward's presentation serves as a call to action for greater transparency, standardization, and collaboration across the industry to build a more accurate and actionable understanding of the cyber threat landscape.

Background

▶ Watch: Introduction: The problem of vendor report bias (0:00)

The cybersecurity industry heavily relies on annual threat reports published by prominent security vendors. These reports, often compiled from vast datasets gathered through incident response engagements, endpoint telemetry, and network monitoring, serve as a primary source of CTI for organizations that lack their own direct pipeline to raw incident data. For many, these reports dictate where security budgets are allocated, which threats are prioritized, and what defensive strategies are adopted. However, the seemingly authoritative nature of these reports masks a fundamental problem: the data they present is often a reflection of the vendor's specific lens, not a comprehensive, unbiased view of the entire threat landscape.

The problem, as Woodward illustrates, begins with data sourcing. Major incident response firms like CrowdStrike, Mandiant, Unit 42, and Verizon collect their data directly from the incidents they investigate. This raw, unfiltered CTI from the "trenches of incident response" is invaluable, as noted by Kevin Mandia years ago. However, the subsequent aggregation and reporting process introduces several layers of potential bias. Each firm has its own client base, often characterized by specific sophistication levels, industry sectors, or financial capacity. For instance, large, sophisticated organizations might retain top-tier IR firms, while smaller entities might opt for less expensive options or be directed by their insurance panels to specific vendors. This creates a self-selection bias, where the incidents reported by a particular vendor might not be representative of the broader industry but rather of their specific client demographic.

Compounding this, the industry lacks a standardized framework for incident reporting, akin to the National Highway Transportation Safety Administration (NHTSA) for vehicle accidents. Without common definitions, reporting mechanisms, and data sharing protocols, each vendor operates within its own silo. This absence of a unified approach means that even when reporting on similar incidents, the interpretation, categorization, and eventual presentation of data can diverge significantly. Woodward's personal journey into this research began with the observation that different reports identified different "number one threat vectors" – a clear indicator that something was amiss in how these critical insights were being generated and consumed. This foundational disagreement underscores the necessity for a deeper understanding of the inherent biases shaping the CTI landscape.

Key Findings

▶ Watch: Conflicting 2024 vendor reports on initial access (4:00)

Woodward's two-year analysis of leading vendor threat reports consistently revealed significant discrepancies in their identification of primary initial access vectors, demonstrating a clear vendor data bias. This divergence was particularly evident when comparing reports covering the 2023 and 2024 periods.

For reports reflecting 2023 data (published in 2024):

• Unit 42 (Palo Alto Networks) identified API vulnerabilities as the number one initial access factor, followed by previously compromised credentials, social engineering (phishing), and brute forcing. Notably, 12% of incidents fell into an "other" category, indicating difficulty in clean categorization.
• Mandiant M-Trends largely agreed, stating exploitation was the number one initial access, followed by phishing, prior compromise, stolen credentials, and brute forcing.
• CrowdStrike's Global Threat Report did not provide raw numbers or percentages but stated that "unmanaged network appliances, particularly edge gateway devices, remain the most routinely observed initial access vector for exploitation." This aligns with exploitation but, due to CrowdStrike's endpoint-centric visibility, primarily captures post-exploitation activities, requiring forensic investigation to trace back to initial edge device compromise.
• Verizon's Data Breach Investigations Report (DBIR), often considered a "gold standard" due to its extensive historical data (approaching 20 years), presented a stark contrast. Verizon reported credentials as the number one vector, with exploits ranking third. This significant disagreement prompted the question: "Who's right?"

The discrepancies persisted in reports reflecting 2024 data (published in 2025):

• Unit 42 shifted, identifying phishing as the number one access vector, followed by software/API vulnerabilities and previously compromised credentials. This raised questions about the categorization of "software/API" versus previous "exploitation."
• Mandiant M-Trends remained consistent, maintaining exploits as the number one vector, followed by stolen credentials and phishing.
• CrowdStrike's Global Threat Report again avoided clear percentages but noted a "boom" in initial access attacks, with a heatmap suggesting exploit for public-facing application and valid accounts were tied for first place. This further highlighted definitional ambiguities.
• Verizon DBIR saw exploitation of vulnerabilities move up to number two, but credential abuse remained number one, introducing another definition challenge: what constitutes "credential abuse"?

Overall trends observed over two years included a potential decline in phishing (suggesting the impact of MFA and phishing-resistant MFA) and a consistent uptrend in exploitation, particularly compared to five years prior. However, the fundamental disagreement on the single "number one" vector across vendors remained. Woodward emphasized that the question isn't which vendor is "wrong," but rather why these different perspectives exist, pointing to a combination of structural factors:

1. Self-Selection Bias: Sophisticated organizations with larger budgets gravitate towards specific high-end IR firms, influencing the types of incidents those firms see.
2. Insurance Panels: Certain insurance companies mandate specific IR firms for their clients, further shaping the client demographics and incident types seen by those firms.
3. Intake Forms: The initial reporting process by victims often lacks standardized taxonomy, leading to inconsistent incident descriptions.
4. Pre-existing Contracts/Retainers: Long-term relationships between clients and IR firms create consistent data pipelines.
5. Reporting Problems (NDAs): Non-disclosure agreements frequently prevent IR firms from publicly sharing detailed findings, leading to "lessons lost" for the broader industry, as highlighted by Daniel Woods et al. at USENIX.
6. Telemetry Differences: Each vendor's core product (e.g., Microsoft's broad telemetry, CrowdStrike's endpoint focus, Wiz's cloud focus) dictates what they can "see" and, consequently, what they report. A cloud security vendor like Wiz, for instance, won't report on Windows registry keys.

These findings collectively illustrate that vendor reports, while valuable, are inherently partial views, shaped by their unique business models, client bases, and technical visibility.

Technical Deep Dive

▶ Watch: Structural reasons for disparate vendor data (10:00)

The core of the problem identified by Eli Woodward lies not in the malicious intent of vendors, but in the structural and definitional challenges inherent in the cybersecurity industry's approach to threat intelligence. This "structural issue," as Woodward calls it, results in disparate data and, consequently, distorted decisions.

1. Data Sourcing and Self-Selection Bias:

The initial intake of incident data is heavily influenced by the client base of each IR firm. Woodward explained that to secure a retainer with firms like Unit 42 or Mandiant, organizations typically need to meet certain sophistication and budgetary requirements. This naturally leads to a self-selection bias, where these top-tier firms primarily service large enterprises, often in specific industries (e.g., financial institutions with a particular vendor). Conversely, smaller organizations or those with different risk profiles might engage other vendors. This means the aggregated data reflects the threats faced by a specific segment of the market, not a universal truth.

Furthermore, insurance panels play a significant role. Insurance companies often have preferred lists of IR firms, channeling incidents from their insured clients towards these specific vendors. This further concentrates certain types of incidents, industries, and attack vectors within particular IR firm datasets, skewing their reported findings.

2. Reporting Limitations and NDAs:

A significant hurdle to holistic threat understanding is the prevalence of Non-Disclosure Agreements (NDAs). Woodward noted the common practice of IR engagements resulting in verbal briefings rather than written reports, or written reports being kept under strict NDA. This directly impacts the ability to share valuable lessons learned with the broader industry. He cited research by Dr. Daniel Woods and colleagues, presented at USENIX, titled "Lessons Lost," which meticulously detailed how NDAs create a "visibility issue" that prevents a true understanding of industry-wide risks. These critical insights, trapped behind legal agreements, cannot inform collective defense strategies, perpetuating a fragmented knowledge base.

3. Telemetry Dictates Visibility:

Each vendor's core product and service offerings dictate its telemetry—what it can actually "see" within an environment.

• CrowdStrike, being endpoint-specific, excels at detecting post-exploitation activity on managed endpoints. While forensics can trace back to initial access, their primary visibility isn't at the network edge or cloud infrastructure where initial compromises often occur.
• Microsoft has vast telemetry across its ecosystem (Windows, Azure, M365), offering a broad but still platform-specific view.
• Wiz, a cloud security platform, focuses almost exclusively on cloud environments. Consequently, their reports will naturally lack insights into on-premise Windows endpoints or registry key activity.
• Mandiant and Unit 42, while having broad IR capabilities, also have specific strengths (e.g., Mandiant's deep expertise in nation-state APTs).

This means that when reading a vendor report, one must consider the inherent limitations of their data collection scope.

4. The Taxonomy Problem:

Perhaps the most fundamental structural issue is the profound lack of standardized taxonomy across the cybersecurity industry. Woodward humorously recounted his master's degree experience, where he could inflate word counts simply by debating definitions. He cited a paper by Sarkar and Shukla, which identified 23 different definitions for "cybercrime"—a staggering example of definitional ambiguity.

This problem permeates CTI. What one vendor classifies as "stolen credentials," another might label "phishing" if the credentials were obtained via a phishing attack, or "brute forcing" if that was the method of compromise. Woodward dissected "credential abuse" as an example, proposing four distinct classifications that would require different data sets and visibility to identify:

• Brute-forcing valid accounts: An active attack.
• Password dumps from prior breaches: Data reuse.
• Info stealers and malware harvesting tokens: Malware-enabled, password-specific breaches.
• Buying credentials from an initial access broker (IAB): An access marketplace compromise.

Each of these requires different investigative techniques and data sources to fully understand the chain of events. Without a common language, comparing and aggregating data across vendors becomes an exercise in interpreting ambiguous terms.

5. The Highway Safety Analogy:

To powerfully illustrate this structural breakdown, Woodward drew an analogy to highway safety. He described a hypothetical model where:

• Police only respond to "really, really, really bad" accidents (like Colonial Pipeline).
• Insurance companies hire their own investigators, who then hide their findings behind NDAs.
• Each investigator uses their own definitions for accident causes.
• Four different companies publish their own "road safety reports," leading to disagreements on basic facts like whether seat belts prevent fatalities.

This chaotic scenario, he argued, perfectly mirrors the current state of cybersecurity reporting, where the lack of a national or international framework prevents a unified understanding of threats and effective, data-driven policy changes.

In essence, the "technical deep dive" reveals that vendor data bias is not a superficial reporting error but a deeply entrenched systemic issue rooted in the fragmented nature of data collection, the competitive landscape of the security industry, and a critical absence of shared definitions and reporting standards.

Demo / Proof of Concept

▶ Watch: Hope for the future: Cyber Incident Reporting Act (CIRCIA) (19:30)

While this talk did not feature a traditional technical demonstration of a tool or exploit, Eli Woodward's multi-year comparative analysis of leading vendor threat reports served as the empirical proof of concept for his hypothesis: that disparate data sources and methodologies lead to distorted threat intelligence. His methodical presentation of conflicting "number one initial access vector" claims across Unit 42, Mandiant, CrowdStrike, and Verizon DBIR reports over two consecutive cycles (2023 and 2024 data) effectively demonstrated the reality of vendor data bias.

The "proof" was in the consistent inconsistencies. By juxtaposing the specific percentages, rankings, and categorical labels used by each vendor, Woodward visually and analytically exposed the problem. For instance, showing how one report prioritized "API vulnerabilities" while another highlighted "credentials" or "exploits" for the same period vividly illustrated the lack of consensus and the underlying definitional and sourcing issues. This analytical demonstration highlighted the urgent need for a more critical approach to consuming CTI.

Furthermore, Woodward briefly touched upon his current work with AI honeypots, noting that he's observing new attack patterns, such as scanners specifically looking for exposed Ollama endpoints, resellers monetizing access, and even confused customers leaking PII into his honeypots. This anecdotal example, though not a formal demo, reinforced the core message that the rapidly evolving threat landscape constantly introduces new attack vectors and requires an equally agile and standardized approach to taxonomy and reporting, underscoring the ongoing relevance and urgency of addressing vendor data bias.

Defensive Implications

▶ Watch: Practical advice for interpreting vendor reports (20:50)

Understanding vendor data bias is not merely an academic exercise; it has profound defensive implications for organizations. CTI teams, security leaders, and CISOs must adapt their strategies to navigate this complex landscape effectively.

1. Triangulate, Don't Cherry-Pick: The most critical defensive strategy is to avoid relying on a single vendor's report. Instead, organizations should triangulate information by consulting multiple sources. This means reviewing reports from a diverse set of vendors (not just the "big four" but also smaller MSPs or industry-specific reports like those from Arctic Wolf or Hunters) to identify commonalities and understand the range of perspectives. This holistic view helps in forming a more balanced and comprehensive understanding of the threat landscape.

1. Match Vendor Profile to Your Organization: Defenders must critically evaluate the vendor's profile and client base against their own organization's characteristics.

• Industry: If a vendor primarily serves financial institutions, their report might be highly relevant to a bank but less so to a small manufacturing firm.
• Size and Sophistication: A report derived from incidents affecting large, highly mature organizations might not accurately reflect the risks for a 500-person company without MFA enabled.
• Technology Stack: Consider what the vendor's core product (endpoint, network, cloud) allows them to see. If your organization is heavily invested in cloud infrastructure, a cloud-centric vendor's report might be more pertinent.
• Insurance Pipeline: Understand which IR firms your insurance provider has on retainer. Their reports might be particularly relevant, and aligning your security investments with their findings can make those purchases more "defensible" (i.e., justifiable to stakeholders and insurance).

1. Question Categorization and Definitions: Always question how vendors define their terms. When a report mentions "credential abuse," ask: Does this include brute-forcing, password dumps, info-stealer malware, or initial access brokers? Each definition implies a different set of defensive controls. For example, if "credential abuse" includes info-stealers, visibility into the stealer marketplace (e.g., via dark web vendor data sources) becomes a critical requirement. This critical inquiry ensures that defensive measures are precisely targeted.

1. Prioritize Phishing-Resistant MFA and Patching: Despite the disagreements, two consistent trends emerged: the potential decline in phishing (suggesting MFA's effectiveness) and the rise in exploitation. Defenders should double down on implementing phishing-resistant MFA (e.g., UbiKeys, FIDO2) across the organization. Concurrently, vulnerability management and faster patching of public-facing applications and edge devices must be top priorities, as exploitation continues to be a prevalent initial access vector.

1. Advocate for Industry Standardization: Organizations should demand better from the industry. This includes advocating for standardized taxonomies and incident reporting frameworks. Emerging legislative efforts like the Cyber Incident Reporting for Critical Infrastructure Act (CIRCIA) in the US and the Cyber Resilience Act (CRA) in Europe represent potential pathways towards more consistent and actionable threat data. Engaging with these initiatives and pushing for broader adoption of common definitions will ultimately benefit all defenders.

By adopting a more critical and informed approach to consuming threat intelligence, organizations can move beyond distorted decisions and make strategic, impactful investments that genuinely enhance their security posture against the threats most relevant to them.

Key Takeaways

• Vendor reports offer partial views: Each vendor's threat report is influenced by their specific client base, telemetry, and business model, providing only a "slice" of the overall threat landscape.
• Taxonomy is a major challenge: The lack of standardized definitions for terms like "cybercrime," "CTI," or "credential abuse" leads to significant confusion and makes cross-vendor comparisons difficult.
• Exploitation is trending up: Across multiple reports, the exploitation of vulnerabilities, particularly in public-facing applications and edge devices, has consistently been identified as a growing initial access vector.
• MFA is making an impact: There's an encouraging trend suggesting a decline in phishing as an initial access vector, likely due to the increased adoption of multi-factor authentication (MFA), especially phishing-resistant variants.
• Triangulate and contextualize CTI: Defenders should consult multiple threat reports, match vendor profiles to their own organization's industry and sophistication, and critically question how data is categorized to form a holistic and relevant threat picture.
• Demand better industry standards: Organizations should advocate for and support initiatives like CIRCIA and CRA to establish standardized taxonomies and reporting frameworks, which are crucial for a more accurate and actionable understanding of global cyber threats.

About the Speaker(s)

Eli Woodward is a Threat Intel Advisor with Team Cymru, where he has been for over a year. Prior to his current role, he spent four years at Zelle, the peer-to-peer financial company in the US. His extensive background also includes significant time in the incident response (IR) world, where he gained first-hand experience dealing with numerous ransomware incidents, particularly during the COVID-19 pandemic. Before entering the private sector, Woodward served for 12 and a half years in law enforcement and government, including a period as a motor cop in Arizona, which deeply ingrained in him the value of standardized reporting frameworks like those used in highway safety. This diverse experience across law enforcement, IR, and financial technology has provided him with a unique perspective on the challenges and importance of accurate and actionable cyber threat intelligence.