Evaluating Threat Intelligence Through Velocity

Evaluating Threat Intelligence Through Velocity

Speakers: Joe Slowik, Paralus

Conference: FIRST CTI Conference

YouTube: https://www.youtube.com/watch?v=lmnZd8wTix8

Overview

In his compelling talk at the FIRST CTI Conference, Joe Slowik of Paralus presented a critical re-evaluation of how cyber threat intelligence (CTI) is assessed, arguing that velocity—the speed of delivery—is as crucial as, if not more important than, traditional metrics like completeness and accuracy, especially for tactical defensive operations. Slowik challenged the prevailing notion that CTI's primary purpose is always to provide exhaustive, long-term strategic insights, instead advocating for a perspective where CTI functions as immediate decision support, enabling rapid, concrete security outcomes.

Slowik emphasized that intelligence, by its very nature, is designed to reduce uncertainty and support decision-making across all organizational tiers, from the C-suite to the Tier 1 Security Operations Center (SOC) analyst. However, the efficacy of this support is inherently tied to its timeliness. Intelligence that arrives too late, after the critical decision window has passed, becomes largely irrelevant, regardless of its inherent quality or depth. This talk underscored the urgent need for CTI producers and consumers to align their processes with the real-world operational speeds of both adversaries and defenders.

The core of Slowik's argument revolved around the fundamental tension between the desire for perfect, comprehensive intelligence and the exigencies of real-time defense. He proposed that for tactical scenarios, a "good enough" answer delivered rapidly often outweighs a "perfect" answer delivered too late. Furthermore, he explored the potential of emerging technologies, particularly generative AI, to help bridge this gap, allowing for faster enrichment and contextualization of threat data without sacrificing necessary detail, thereby offering a path to reconcile the competing demands of speed and accuracy in CTI production.

Background

▶ Watch: Introduction: Evaluating threat intelligence through velocity (0:00)

Slowik initiated his discussion by questioning the fundamental purpose of CTI, even within a dedicated threat intelligence conference. He posited that while many define CTI as a shift from reactive to proactive security, its most critical role is decision support. Intelligence, in this context, serves to reduce uncertainty for stakeholders ranging from SOC analysts to CISOs, enabling better decisions in imperfect information environments. From this foundation, Slowik asserted that intelligence not supporting a decision, or one that doesn't lead to a better decision, holds questionable value.

While CTI can serve strategic, long-term planning, Slowik argued that its greatest impact on defending an organization lies in immediate tactical defense and response. He highlighted that for "applied threat intelligence," the goal is to translate insights into specific, concrete security outcomes. This focus on tactical utility immediately brings to light inherent tensions in CTI delivery, often framed by the "good, cheap, fast" dilemma:

• Good and Cheap: Not fast.
• Fast and Good: Not cheap.
• Cheap and Fast: Not good.

This analogy illustrates the spectrum of CTI products. At one extreme are indicators of compromise (IOCs) or indicators of attack (IOAs)—terse, simplistic technical observations shared rapidly with minimal investment. At the other are exhaustive, 20-page formal intelligence reports detailing threat actor operations over years, requiring significant time and resources. Slowik stressed that the optimal position along this continuum is dictated by the audience and the decisions they need to make. A Tier 1 SOC analyst, operating within tight time windows, does not need a lengthy white paper, just as a C-suite executive would find a raw list of IOCs largely useless.

Crucially, Slowik argued that CTI alerts must arrive oriented to the time horizons of those decisions. Intelligence delivered too late, or after the decision point, becomes irrelevant. He drew parallels to security operations metrics like Mean Time to Respond (MTTR) and Mean Time to Closure (MTTC), emphasizing that CTI must align with these workflows and the speed at which threats operate. Therefore, evaluating CTI's effectiveness must extend beyond completeness and accuracy to include the time investment required, necessitating a trade-off: is it better to have a 100% complete answer next week, or a 70% complete answer within the next hour that an incident response team can immediately act upon?

Key Findings

▶ Watch: Greatest impact: Tactical CTI for immediate defense (4:50)

Joe Slowik's presentation distilled several critical findings regarding the nature and evaluation of cyber threat intelligence:

• CTI as Decision Support: The fundamental purpose of CTI is to reduce uncertainty and enable better decision-making across all organizational levels. Intelligence that fails to support a decision or improve an outcome is of questionable value.
• Velocity is a Critical Metric: For tactical defensive operations, the speed of CTI delivery, or its velocity, is as important as its accuracy or completeness. Timely intelligence, even if imperfect, is superior to perfect intelligence delivered too late.
• Tactical Focus for Maximum Impact: CTI achieves its greatest effect when oriented towards immediate tactical defense and response, translating insights into discrete, actionable security outcomes.
• The Speed vs. Accuracy Trade-off: There is an inherent tension in CTI production between achieving maximal completeness and accuracy (which requires time) and delivering actionable intelligence rapidly. Organizations must consciously manage this trade-off based on stakeholder needs.
• Alignment with Operational Timelines: Effective CTI must be delivered within the operational time horizons of both defenders (often hours or minutes) and adversaries, ensuring it can expedite defensive responses before adversary objectives are achieved.
• AI as an Enabler: Emerging technologies, particularly generative AI, offer a promising avenue to reconcile the tension between speed and accuracy. AI can rapidly enrich and contextualize data, augmenting human analysts and accelerating CTI production without replacing human cognition.

Technical Deep Dive

▶ Watch: Tensions in CTI delivery: Speed versus depth (6:10)

Slowik moved from the theoretical underpinnings of CTI's purpose to its practical application in the context of adversary operations, which he characterized as focusing on actions and objectives. Adversaries infiltrate networks to achieve specific goals, whether it's holding systems for cryptocurrency, exfiltrating sensitive data, or deploying destructive payloads. CTI's interjection point, ideally, is prior to adversaries achieving these objectives.

He outlined a typical intrusion lifecycle:

1. Initial Foothold: Through phishing, exploiting external assets, etc.
2. Lateral Movement and Defense Evasion: Expanding presence, establishing persistence.
3. Actions on Objectives: Mission accomplishment (e.g., data theft, system disruption).

The critical insight here is that defense "wins" so long as it can interrupt adversary progress before they achieve their mission. This doesn't necessarily mean stopping them at initial access, though that's ideal. It means CTI must expedite defensive response, allowing security teams to adapt to evolving adversary tradecraft and co-evolve their defenses.

Slowik provided theoretical use cases to illustrate how CTI velocity applies to different threat scenarios:

• E-crime and Ransomware Ecosystems: For opportunistic threats, rapid CTI enables preventative actions. This includes advocating for patching newly discovered vulnerabilities (plus one day vulnerabilities) or identifying emerging techniques (e.g., click-fix actions) to close gaps before intrusions occur. Here, CTI helps get ahead of the adversary's initial access.
• Advanced Persistent Threat (APT) and Espionage Operations: These adversaries often engage in low-and-slow intrusion operations, aiming for long-term presence to continuously collect and exfiltrate data. CTI's role here is to rapidly identify persistence mechanisms and exfiltration pathways. Even if initial defenses are bypassed, timely CTI can detect the breach and mitigate harm by identifying how data is leaving the environment.
• State-Sponsored Disruption and Destructive Events: These pose existential risks to organizations, often involving wiper malware or similar payloads. Here, time is critically short. CTI must rapidly share trends in threat behaviors and identify adversary pre-positioning for effects (e.g., compromising Active Directory, developing wormable functionality). The goal is to interject before the destructive payload is successfully deployed, winning the race against the adversary's final action.

Across these scenarios, Slowik emphasized that adversary and defender timelines are often measured in hours, sometimes even minutes. This stark reality underscores why leisurely CTI production is no longer viable. He reiterated that CTI that arrives too late is useless for operational outcomes, advocating for pushing results as early as possible into stakeholders' hands, even if it means sacrificing some completeness.

However, Slowik also acknowledged exceptions where speed is not the primary driver:

• Strategic Support: C-suite decisions on long-term security program maturation (e.g., 5-year investment plans) require comprehensive, accurate answers, not rapid, partial ones.
• Preventative Controls: Activities like attack surface management and patch management prioritize completeness and understanding the full state of the environment over immediate speed.
• Defensive Preparation and Training: Informing SOC analysts, incident responders, and threat hunters about adversary environments for training and preparedness benefits from detailed, rigorous analysis.

This led to the concept of two discrete tracks for cyber threat intelligence:

1. Rapid, Tactical Support: Focused on "threat indications and warning," providing immediate support for defensive decision-making.
2. Higher-Level, Detailed Support: Aligned with strategic decisions, producing in-depth reports and multi-source analysis.

The challenge with this bifurcated approach, Slowik noted, is complexity and cost. For most organizations, CTI is already a "luxury item," and funding two distinct CTI functions is often unfeasible.

This is where Slowik introduced the potential of artificial intelligence (AI), specifically generative AI, as a solution to "square the circle" – achieving both speed and accuracy. Initially a skeptic, Slowik's perspective shifted as AI technology evolved. He cautioned that generative AI is still early-stage and not a replacement for humans. Instead, he sees it as an augmentation for human-driven workflows. By asking the right questions, designing effective workflows, and building robust decision pipelines, AI can rapidly enrich and enhance data context. Whether through a more time-constrained deep research agent or a quicker large language model (LLM), AI can add those crucial steps of context and detail that human analysts simply lack the cycles to provide at speed.

Slowik concluded this section by stressing that the future of CTI is technologically assisted, not replaced. Organizations that fail to critically consider how AI can accelerate CTI production and enhance accuracy risk an evolutionary disadvantage against adversaries who are already keen to leverage any available tool to increase their own velocity of attack cycles. The challenge for the CTI community is to upskill and leverage these tools to deliver more complete answers rapidly, enabling stakeholders to respond before intrusions are fully manifested.

Demo / Proof of Concept

▶ Watch: CTI's role in adversary and defender timelines (13:00)

While Joe Slowik's presentation did not include a live technical demonstration or a software proof of concept, he explicitly framed his discussion around "theoretical use cases" to illustrate the impact of CTI velocity. He acknowledged the difficulty in providing concrete, real-time examples due to the rapid pace of threat evolution. Instead, he leveraged conceptual adversary operations and defensive timelines to demonstrate how timely CTI intervenes in the intrusion lifecycle for various threat types, such as e-crime, APT, and state-sponsored destructive attacks, thereby serving as a robust conceptual proof of the principles he advocated.

Defensive Implications

▶ Watch: Leveraging AI to balance CTI speed and accuracy (22:00)

The implications of Joe Slowik's talk for defensive security teams are profound, urging a fundamental shift in how CTI is consumed, produced, and integrated into security operations:

• Prioritize Timeliness for Tactical CTI: Defenders must recognize that for immediate incident response and threat mitigation, the speed of CTI delivery is paramount. A "good enough" answer delivered in minutes or hours is often more valuable than a perfect, exhaustive report delivered days later.
• Define and Align with Operational Time Horizons: Security teams, particularly SOC and IR, need to clearly define their decision-making time windows. CTI producers must then tailor their output and delivery mechanisms to align with these critical windows, ensuring intelligence arrives when it can have maximum impact.
• Integrate CTI with Operational Metrics: Just as security operations are measured by metrics like Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR), CTI's effectiveness should also be assessed by its velocity and impact on these operational timelines.
• Embrace "Good Enough" for Rapid Response: Teams should become comfortable with acting on intelligence that may not be 100% complete or perfectly contextualized if it enables a faster, more effective defensive action against an active threat. This requires a cultural shift towards risk-informed decision-making.
• Investigate and Adopt AI/ML for CTI Acceleration: Organizations should actively explore and integrate generative AI and other machine learning technologies into their CTI workflows. These tools can serve as powerful augmentations for human analysts, rapidly enriching data, generating initial analyses, and synthesizing information to accelerate delivery without sacrificing critical detail.
• Consider Tiered CTI Products: For larger organizations, developing a bifurcated CTI strategy might be necessary: one track focused on rapid, terse indications and warnings for operational teams, and another for detailed, strategic intelligence for long-term planning and executive decision-making. If full bifurcation isn't possible, CTI teams must master tailoring existing products to different audiences with varying time sensitivities.
• Focus CTI on Disrupting Adversary Objectives: Defensive efforts, supported by CTI, should aim to interrupt the adversary's progress at any point before they achieve their final objectives, not just at initial access. This means CTI needs to provide insights into lateral movement, persistence, and pre-positioning activities with high velocity.
• Upskill CTI Analysts and Security Teams: Analysts need to be trained not only in traditional intelligence analysis but also in leveraging new technologies like AI to enhance speed and efficiency. Security teams, in turn, need to understand how to effectively consume and act upon rapidly delivered, potentially less complete, intelligence.

Key Takeaways

• CTI's core value is decision support: Its primary purpose is to reduce uncertainty and enable better, faster decisions for security outcomes, from the SOC to the C-suite.
• Velocity is paramount for tactical CTI: For immediate defensive actions, the speed of intelligence delivery is as critical as its accuracy or completeness; untimely intelligence is irrelevant.
• Align CTI with operational realities: CTI must be delivered within the compressed time horizons (hours, minutes) of both adversaries and defenders to effectively disrupt attacks and expedite response.
• Embrace the speed-accuracy trade-off: Organizations must consciously balance the desire for perfect, exhaustive intelligence with the need for rapid, actionable insights, prioritizing the latter for immediate tactical situations.
• AI will augment CTI capabilities: Generative AI and other advanced technologies offer a promising path to reconcile the tension between speed and accuracy, enabling faster enrichment and contextualization of threat data without replacing human analysts.
• Adapt or face disadvantage: Failing to evolve CTI production and delivery speeds, particularly through technological assistance, will leave organizations at an evolutionary disadvantage against rapidly operating adversaries.

About the Speaker(s)

Joe Slowik is affiliated with Paralus, where he contributes his expertise in cyber threat intelligence. Throughout his career, Slowik has demonstrated a strong focus on applied threat intelligence, advocating for CTI that directly translates into concrete security outcomes and decision support. He is also involved in teaching, sharing his insights on how to effectively operationalize threat intelligence. Slowik is known for his analytical approach and willingness to challenge conventional wisdom, as evidenced by his evolving perspective on the role of artificial intelligence in CTI, moving from initial skepticism to recognizing its significant potential as an augmentation tool for human analysts.