Stop the flood: building a quality and trust-driven threat intelligence ecosystem

Stop the flood: building a quality and trust-driven threat intelligence ecosystem

Speakers: Kihong Kim (CEO and founder, SANDS Lab); SuhMahn Hur (SANDS Lab)

Conference: Virus Bulletin

YouTube: https://www.youtube.com/watch?v=5KilfG_eQ_g

Overview

In the rapidly evolving landscape of cyber threats, Threat Intelligence (TI) has become an indispensable component of defensive strategies. However, the sheer volume and often questionable quality of available threat intelligence data present significant challenges for security professionals. This talk, "Stop the flood: building a quality and trust-driven threat intelligence ecosystem," delivered by Kihong Kim, CEO and founder of SANDS Lab, and SuhMahn Hur, directly addresses these critical issues. The speakers highlight how an overwhelming influx of low-value, unverified Indicators of Compromise (IOCs) is rendering security teams ineffective, preventing them from making timely and accurate decisions.

The core problem articulated is a pervasive disconnect between the massive quantity of threat data and its actual utility. Many providers flood the ecosystem with IOCs that lack sufficient context, reliability, or relevance, leading to a high rate of false positives and an inability for security personnel to properly process and respond. SANDS Lab, through its own experience, manages between 100,000 and 500,000 detection approvals, yet identifies a high false positive rate of 0.0022% with raw data, underscoring the urgency of a more refined approach. This presentation details the inherent problems with current threat intelligence practices and proposes a novel, provider-centric trust model to cultivate a more reliable and actionable threat intelligence ecosystem.

The talk is crucial for anyone involved in cybersecurity operations, threat hunting, or strategic security planning. It provides a sobering look at the current state of threat intelligence consumption and offers practical insights and a forward-thinking framework for improving the efficacy of TI. By focusing on quality and trust, the proposed model aims to transform the deluge of data into genuinely actionable intelligence, thereby empowering defenders to proactively counter cyber threats with greater precision and confidence.

Background

▶ Watch: Introduction: Overwhelmed by low-quality threat intelligence data. (0:00)

The genesis of this problem lies in the explosion of threat data coupled with a lack of standardized quality control. Surveys indicate that 61% of security teams are overwhelmed by the enormous amount of data, while 59% struggle to verify its validity and relevance. This "flood" of information, often of low quality, makes effective interpretation and response nearly impossible for cybersecurity professionals. To understand this challenge, it's essential to define what constitutes "intelligence" in this context. The speakers categorize intelligence into three main types:

• Strategic Intelligence: High-level information based on global events, geopolitical situations, and organizational risks, used by executives for strategic decision-making.
• Operational Intelligence: Focuses on the "who, what, and how" behind attacks, dealing with threat actors, their motivations, and methodologies.
• Tactical Intelligence (IOCs): High in nature and immediate, dealing with specific indicators of compromise like malicious URLs, file hashes, domains, and IP addresses. This talk primarily focuses on IOCs.

The Pyramid of Pain framework illustrates that IOCs, while easy to obtain and frequently used, often possess a relatively low information value. They are the easiest artifacts for attackers to change, meaning their shelf life can be short. IOCs can be collected from various sources, including internal infrastructure (e.g., honeypots) or external feeds. However, the speakers emphasize that incoming IP addresses, for example, can be a mixture of simple scanning IPs, infiltration IPs, or C2 (Command and Control) IPs. While all are potentially malicious, simple scanning IPs might originate from legitimate sources like Autonomous System Numbers (ASNs) or sandboxes, making them less indicative of an actual threat than an infiltration or C2 IP.

The utility of collected IOCs also varies significantly depending on their origin and type. IOCs collected from honeypots — decoy systems designed to attract and trap attackers — prove valuable across all incident handling stages. In contrast, IOCs obtained from broader Information Sharing and Analysis Centers (ISACs) or general CTI (Cyber Threat Intelligence) feeds often have limited utility, primarily for detection and less so for deeper analysis or response. This highlights a critical disparity in data value based on the collection method and inherent reliability.

Beyond general utility, the talk delves into specific issues of IOC locality and reliability. A compelling case study involved a phishing SMS campaign targeting various regions globally. While the campaign used common tactics like APK downloads and personal information harvesting, the specific phishing domains and underlying infrastructure (e.g., unique IPs) were localized. For instance, Korean targets received phishing domains using specific IPs, while Indonesian targets received different ones. This demonstrates that even if an IOC is "malicious," its relevance might be geographically constrained, making global application without context inefficient.

Another significant issue arises from the reliability of shared intelligence. The Korea Internet & Security Agency (KISA) operates a national threat intelligence service where members share IOCs. However, the speakers recount an incident where shared data appeared anomalous, with many reported IPs turning out to be normal or randomly generated upon investigation. This directly undermines trust and highlights the challenge of blindly ingesting data from even reputable sources without robust validation mechanisms. Existing efforts like MISP (Malware Information Sharing Platform) have attempted to address these issues by introducing concepts like a decay model for IOC relevance and various classification tags, including the Admiralty Scale for source reliability and information credibility. While these are valuable frameworks, their global, consistent application remains a significant hurdle.

Key Findings

▶ Watch: Real-world issues: Unreliable shared IOCs from a national agency. (8:00)

The central finding of this presentation is the critical need to shift from a quantity-driven to a quality and trust-driven threat intelligence ecosystem. The current "flood" of IOCs, characterized by low information value and questionable reliability, is not only overwhelming security teams but actively hindering effective cyber defense. The speakers demonstrate that traditional methods of evaluating IOCs based on their individual characteristics are profoundly difficult and resource-intensive to implement consistently across the vast landscape of available data.

Key findings include:

• Overwhelming Data Volume: A significant majority of security professionals (61%) are overwhelmed by the sheer volume of threat intelligence data, with nearly 60% struggling to verify its validity and relevance.
• Low Intrinsic Value of Raw IOCs: While easy to obtain, raw IOCs often have low information value and limited utility across the full incident response lifecycle, especially when compared to high-fidelity IOCs derived from active attacker engagement (e.g., honeypots).
• Challenges in IOC-Based Reliability Evaluation: Despite frameworks like MISP's decay model and Admiralty Scale, assigning consistent reliability and quality scores to each individual IOC is a "very difficult work" that requires universal commitment, which is rarely achieved.
• Minimal Adoption of STIX 2.1 Confidence Scores: Even with the introduction of new objects and a confidence score (0-100) in STIX 2.1 for data reliability, its actual usage by data providers in the CTI ecosystem has been minimal, indicating a gap between standard capabilities and practical implementation.
• Proposed Provider-Based Trust Model: The most significant contribution is the proposal of an evaluation model based on provider trust scores (ranging from 0 to 1000 points) rather than solely on individual IOC data reliability. This model considers factors like legislation compliance, data update frequency, peer review content, and user evaluations.
• Dramatic Improvement in Detection Efficacy: By implementing internal processes that prioritize quality, such as collecting liability for collection sources and evidence for IOC data, SANDS Lab was able to reduce their valid data from 100,000-500,000 items to 6,000-12,000 items. This refined set yielded a detection rate of 0.0074%, which is 34 times higher than the previous 0.0022% when considering the total volume, demonstrating the profound impact of a quality-first approach. This highly valuable figure does not involve active "hunting" but rather focuses on refining the input.

These findings collectively point towards a paradigm shift: instead of trying to filter a deluge of undifferentiated data, the focus should be on building trust at the source, ensuring that the intelligence ingested is inherently more reliable and actionable.

Technical Deep Dive

▶ Watch: Existing solutions: MISP decay model and ADM framework. (9:40)

The speakers delved into existing and proposed technical mechanisms for improving threat intelligence quality. One of the long-standing efforts to address these issues comes from MISP (Malware Information Sharing Platform). MISP recognized the challenges of mass IOC sharing and introduced systems like a decay model to manage the relevance and validity of IOCs over time. This model uses a base score for each IOC and applies a decay function based on its lifetime and decay speed to calculate a final score, reflecting its diminishing relevance.

Furthermore, MISP utilizes tags to provide critical context. For instance, users can "thumb up" an IOC if they actually observe it or "thumb down" if it's a false positive. More formally, MISP incorporates the Admiralty Scale, a widely recognized system for evaluating the reliability of intelligence sources and the credibility of the information itself. This framework assesses two factors:

• Source Reliability (A-F): Rates the trustworthiness of the information source (A being completely reliable, F being unreliable). This is evaluated separately from the data, considering the source's history, skill, and relevance.
• Information Credibility (1-4): Rates the validity and consistency of the information (1 being confirmed by other independent sources, 4 being doubtful).

While these MISP features provide robust tooling, the speakers highlight the practical difficulty of their consistent global application. Manually assigning classifications and reliability scores to each piece of IOC data is an arduous and often unscalable task.

SANDS Lab's internal approach to generating high-reliability IOCs involves honeypots and deception technology. They actively induce actual attacker infiltration, collecting detailed evidence of attacker actions. This includes confirming account information entered by attackers and shell commands executed by them. Such information, backed by clear evidence, is assigned a high risk level, ensuring a strong foundation for their internal threat intelligence. This method contrasts sharply with passively collected IOCs that lack direct evidence of malicious intent or active exploitation.

The talk also examined the current status and development direction of CTI (Cyber Threat Intelligence) sharing, specifically focusing on STIX 2.1 (Structured Threat Information Expression). STIX 2.1 is a widely adopted standard for automated threat intelligence sharing, comprising SDOs (STIX Domain Objects) representing entities like campaigns, malware, or indicators, and RDOs (STIX Relationship Objects) representing relationships between them. A key update from STIX 2.0 to 2.1 was the introduction of new objects and concepts for data reliability. Specifically, note and opinion objects were added for inputting comments and opinions about data, and a new score concept called confidence was introduced. This confidence score, a numerical value from 0 to 100, is intended to be calculated and provided by the data-sharing companies themselves, indicating their level of certainty in the data.

However, the speakers' analysis of STIX 2.1 usage in the CTI ecosystem revealed that the confidence score has been minimally utilized. Their statistical analysis of CTI data, excluding relationship objects, showed even data collection over time, with some spikes. A notable finding from their ChatGPT-summarized analysis (a meta-observation on how they processed the data) was the scarcity of campaign and vulnerability information, suggesting that while there were over 800,000 relationship objects, their meaning might often be at a simple connection level rather than providing deep contextual insights into threat actors or their motivations. They provided an example of a common data pattern: specifying a detection based on a specific port over sources on the left side, and a specific IP on the right side, confirming numerous such basic data cases.

Recognizing the inherent difficulties in IOC-based reliability evaluation, the speakers proposed a novel provider-based evaluation model. This model assigns a provider trust score ranging from 0 to 1000 points, evaluated by various factors:

• Legislation compliance: Adherence to relevant data sharing and privacy regulations.
• Data update frequency: How regularly the provider updates their intelligence.
• Peer review content: Quality and quantity of reviews from other users.
• Initial CTI start level: A baseline trust level.
• User evaluation: Periodic evaluations by data consumers.

This system allows providers to identify and address weaknesses in their databases based on feedback. The provider's reliability score is calculated by weighting each item, considering review data, subscription rates, evaluation volume, and update frequency. These trust levels can then be used by both providers (to improve) and users (to filter). For instance, if an IOC is detected by only one low-trust provider, users might not trust that data. This shifts the burden of trust from individual IOCs to the entities that curate and provide them, fostering a more accountable and transparent ecosystem.

Demo / Proof of Concept

▶ Watch: Proposed solution: A provider-based trust score evaluation model. (18:50)

The talk primarily focused on analysis and a proposed model rather than a live demonstration of a specific tool or attack. However, the speakers detailed SANDS Lab's internal processes for generating high-reliability IOCs, which serves as an internal proof of concept for their methodology. They described using honeypots and deception technology to induce actual attacker infiltration. This allowed them to collect concrete evidence, such as actual account information entered by attackers and shell commands executed on their systems. This "evidence-generated" intelligence is then managed by assigning a risk level, with information backed by clear evidence (like direct attacker interaction via a shell) receiving a "high" risk level. This internal system exemplifies how high-fidelity, evidence-based IOCs can be generated and prioritized, contrasting with the general flood of unverified data. While not a live demo for the audience, it illustrates the practical application and benefits of their quality-driven approach.

Defensive Implications

▶ Watch: New model's impact: 34x higher valid detection rate. (21:30)

The insights shared in this talk carry significant implications for cybersecurity defenders, urging a critical re-evaluation of current threat intelligence consumption strategies.

1. Prioritize Quality Over Quantity: Defenders must resist the urge to ingest every available IOC feed. Instead, focus on sources known for their high quality, context, and verifiable evidence. Blindly integrating massive volumes of low-fidelity IOCs only increases alert fatigue and false positives.
2. Implement Internal Validation: Organizations should develop internal mechanisms to validate incoming threat intelligence. This could involve cross-referencing IOCs with internal telemetry, conducting sandbox analysis, or verifying against existing threat actor profiles. The KISA example highlights the danger of trusting data without verification.
3. Leverage Deception Technology and Honeypots: Investing in and deploying honeypots and deception technology can provide highly valuable, first-party IOCs with undeniable evidence of malicious activity. These internal sources often yield the most actionable intelligence for specific organizational contexts.
4. Advocate for Provider-Based Trust: Defenders should encourage their threat intelligence providers to adopt and publicly share their provider trust scores. As consumers, demanding transparency on data quality and source reliability can drive the industry towards a more trustworthy ecosystem.
5. Contribute to STIX 2.1 Confidence Scores: If utilizing STIX 2.1, defenders should actively use and contribute to the confidence score and note/opinion objects when sharing or receiving intelligence. This helps build a richer, more nuanced understanding of data reliability across the community.
6. Consider Regional and Contextual Relevance: Not all IOCs are relevant to every organization. Defenders should filter and prioritize intelligence based on geographical relevance, industry sector, and specific threat actor targeting their organization, as demonstrated by the localized phishing campaigns.
7. Refine Detection Rules: By focusing on high-fidelity IOCs, defenders can refine their detection rules, making them more precise and reducing false positives. The SANDS Lab finding of a 34x higher detection rate with quality-filtered data underscores this potential. This allows security teams to focus on genuine threats rather than sifting through noise.
8. Educate Teams on TI Limitations: Security teams need to be educated on the inherent limitations of raw IOCs and the importance of contextualizing intelligence. Understanding the "Pyramid of Pain" helps in setting realistic expectations for different types of threat data.

By adopting these defensive strategies, organizations can transform their threat intelligence operations from a reactive, overwhelming struggle into a proactive, efficient, and highly effective defense mechanism.

Key Takeaways

• The current threat intelligence ecosystem is overwhelmed by a massive volume of low-quality, unverified Indicators of Compromise (IOCs), leading to alert fatigue and ineffective security operations.
• Traditional methods of evaluating IOC reliability on an individual basis (e.g., through MISP's decay model or Admiralty Scale) are difficult to implement consistently and globally.
• SANDS Lab's internal experience shows that by focusing on quality and evidence-backed IOCs (e.g., from honeypots), they achieved a 34 times higher effective detection rate compared to raw, unfiltered data.
• The proposed solution advocates for a provider-based trust score model, evaluating threat intelligence sources based on factors like legislation compliance, update frequency, peer reviews, and user evaluations, rather than solely on individual IOC attributes.
• Despite its introduction in STIX 2.1, the confidence score for data reliability is minimally utilized, indicating a gap between standard capabilities and practical adoption in the CTI sharing community.
• Defenders should prioritize quality over quantity, implement internal validation mechanisms, leverage deception technology, and advocate for greater transparency and trust in their threat intelligence providers to build a more effective defense.

About the Speaker(s)

Kihong Kim is the CEO and founder of SANDS Lab, where he also leads the analysis team. His work focuses on addressing the critical problems arising from the flood of low-quality threat intelligence data and developing solutions to enhance its efficacy.

SuhMahn Hur is also associated with SANDS Lab, contributing to the research and development of robust threat intelligence practices.