Billions of Indicators, Zero Action: How We Fixed That

Billions of Indicators, Zero Action: How We Fixed That

Speakers: Daniel Lima (NTT DATA), Gabriel Testoni (NTT DATA)

Conference: FIRST CTI Conference

YouTube: https://www.youtube.com/watch?v=i9Ai0x4ERNE

Overview

In an era defined by an overwhelming deluge of cybersecurity data, security teams often find themselves "drowning by data, but starving for actionable outcomes." This critical challenge was the focus of Daniel Lima and Gabriel Testoni's presentation at the FIRST CTI Conference, titled "Billions of Indicators, Zero Action: How We Fixed That." The talk meticulously dissects the pervasive problem of intelligence operationalization, highlighting that less than 10% of the billions of indicators of compromise (IOCs) processed globally ever translate into concrete defensive measures.

The core of Lima and Testoni's discussion revolves around bridging the significant "intelligence to action gap" that plagues modern security operations centers (SOCs). They argue that while organizations have mastered data collection, the latency between ingesting threat intelligence and enacting defensive actions remains a critical bottleneck. The speakers, representing NTT DATA's experience running four SOCs across Latin America, present a modular and automated Cyber Threat Intelligence (CTI) pipeline designed to transform static, high-volume reports into smart enrichment and feedback loops. Their solution aims to empower security teams at any maturity level to move beyond mere reporting and achieve genuine, proactive protection.

Background

▶ Watch: The 'intelligence to action' gap: 90% of IOCs wasted (4:00)

The current state of cyber threat intelligence, as articulated by the speakers, is characterized by a fundamental disconnect between data acquisition and defensive action. Gabriel Testoni starkly points out that 90% of collected IOCs never trigger any defensive measure, representing a monumental waste of effort. The mean time to block a threat in a typical environment can stretch to four to six hours, a critical window during which attackers can achieve lateral movement, data exfiltration, or establish persistence. Compounding this inefficiency, an alarming 40% of an analyst's valuable time is consumed by mundane data entry tasks—copying and pasting information between disparate tools—rather than actual analysis, hunting, or strategic thinking. This "intelligence to action gap" is the central problem their talk addresses.

Testoni identifies three primary bottlenecks that prevent the traditional CTI cycle (collection, analysis, dissemination) from culminating in effective action. First, silos create fractured security ecosystems where threat intelligence platforms don't actively communicate with SIEMs, and EDRs remain isolated from SOAR solutions. Different vendors, data models, and APIs render data useless, preventing a holistic view and coordinated response. Second, manual processes force highly skilled analysts into repetitive, low-value data movement tasks, squandering human intuition on activities that should be automated. This leads to inefficiencies and delays. Third, alert fatigue is an insidious problem stemming from the continuous addition of CTI feeds without adequate context or prioritization. The resulting deluge of low-trust data overwhelms analysts, leading to desensitization and, often, a complete cessation of action. In essence, silos impede data flow, manual processes hinder speed, and fatigue compromises the quality of defensive measures. The speakers challenge the audience with a poignant question: "Does your security actually really protect or just produce reports?" asserting that true intelligence, by definition, must drive action, not merely documentation.

Key Findings

▶ Watch: Automating enrichment: Raw data to actionable high-confidence intelligence (8:40)

The core finding presented by Daniel Lima and Gabriel Testoni is that the intelligence-to-action gap can be effectively bridged through a modular and automated CTI pipeline built on open-source tools and a continuous feedback loop. Their architecture for action is structured around five interconnected stages: Collect, Process, Enrich, Disseminate, and Action. A critical innovation is the explicit incorporation of a feedback loop, where SOC response data, including human decisions and automated metrics, directly feeds back into the pipeline, allowing the system to learn and improve autonomously over time.

A foundational principle guiding their entire architecture is that "every arrow in this pipeline must be an API call or a script." This commitment eliminates human intervention in data movement, reserving human expertise for critical decision-making. By leveraging automation, the system achieves unprecedented speed and accuracy, transforming raw data into high-confidence, actionable intelligence. The speakers demonstrate that this approach not only drastically reduces the time to block threats but also significantly lowers false positive rates, restoring confidence in security alerts and ultimately enabling security teams to shift from a reactive to a proactive defensive posture. Their findings underscore that high-level security does not necessitate high-cost commercial platforms, as robust open-source components can form the backbone of a highly effective and maintainable system.

Technical Deep Dive

▶ Watch: Tangible results: Drastically improved speed and reduced false positives (16:00)

The NTT DATA team's solution is an architecture for action built on five core stages: Collection, Processing, Enrichment, Dissemination, and Action, all underpinned by a continuous feedback loop. The entire system is engineered with the principle that every interaction is an API call or a script, ensuring no human intervention is required for data movement. This allows humans to focus on decision-making, while AI and automation handle the heavy lifting.

The Collection phase is designed with modular connection at its heart. Each data source, whether it's OSINT feeds, infrastructure monitoring, or dark web forums, operates as an independent script, akin to a microservice or "LEGO bricks." This modularity ensures resilience; if one source's API changes or a forum goes offline, only that specific "brick" needs to be updated or replaced, without disrupting the entire pipeline. This contrasts sharply with monolithic systems, which are prone to cascading failures.

Following collection, the data undergoes Automated Enrichment. This is where raw, often meaningless, data points are transformed into actionable intelligence. For instance, a simple raw IP address, initially "meaningless," is automatically passed through a series of enrichment steps:

1. VirusTotal scores are queried to assess its reputation.
2. Whois records are reviewed for registration details.
3. GeoIP data pinpoints its geographical location.
4. IRM (Internal Risk Management) data is checked for internal context.
5. Metasploit/Cobalt Strike signatures are cross-referenced to identify known malicious infrastructure.

Through this 100% automated process, that "meaningless" IP address can rapidly be identified as a "high confidence Cobalt Strike C2 server," ready for immediate defensive action. No analyst interaction is required during this critical transformation.

The central nervous system of this operation is the Orchestration layer, dubbed "the brain." Moving away from monolithic security platforms, the team built a hub-and-spoke model. The hub, the orchestration layer itself, is powered by simple Python scripts and leverages robust open-source tools such as Shovel and Nmap (likely for scanning and analysis components, rather than the core orchestration engine itself). The spokes represent the entire security ecosystem, including threat intelligence platforms, SIEMs, EDRs, IR scanners, and even notification platforms. A key takeaway here is that sophisticated security capabilities do not necessitate expensive commercial platforms; open-source components can be leveraged to build a cost-effective, debuggable, and maintainable system.

Dissemination is reimagined as an API call, not an email. This is crucial for achieving machine-speed defense. The intelligence is disseminated based on confidence levels determined by the system's internal scoring engine.

• High Confidence IOCs, such as confirmed Cobalt Strike C2s or malicious hashes, are automatically pushed directly to the EDR and firewall for immediate blocking. This tier requires no human approval, as the system's vetting process is deemed sufficient.
• Medium Confidence Indicators generate alerts within the SIEM, requiring an analyst's review and judgment to determine further action or gather more information.
• Strategic Trend Level Intelligence flows upwards to dashboards, providing visibility for leadership and informing broader security strategy.

This tiered approach is a hallmark of operational maturity, balancing the need for rapid, automated blocking of confirmed threats with human oversight for ambiguous cases.

Finally, the Feedback Loop is the "secret weapon" that enables the system to be self-healing and continuously improve. SOC response data, encompassing both human decisions (e.g., whether an alert was a true incident or a false positive) and automated metrics, feeds directly back into the orchestration layer.

• If a CTI source consistently generates noisy or irrelevant indicators (resulting in false positives), its trust score gradually fades, and the system may eventually drop it entirely.
• Conversely, if an indicator leads to a confirmed incident, the system not only increases the trust score of the originating source but also pivots to find related indicators from the same campaign, enhancing its predictive capabilities.

The SOC's daily investigation process thus becomes the training data for the CTI pipeline, eliminating the need for extra funding or manual updates to improve the system's intelligence. This continuous learning mechanism allows the intelligence to "grow every click."

Demo / Proof of Concept

▶ Watch: Secure budget by reporting business impact: analyst hours saved (18:50)

The speakers illustrated the practical application and impact of their automated CTI pipeline through three compelling case studies, demonstrating its efficacy across different operational scenarios.

Case 1: Proactive Phish Hunting

This case study highlights a shift from reactive to proactive defense against phishing attacks. Instead of waiting for users to report suspicious emails, the automated pipeline actively monitors certificate transparency logs via tools like CertStream. It performs analysis on permutations of brand keywords. The moment a suspicious domain, such as "my-dash-bank.com," is registered, the pipeline triggers an alert. Automatically, it takes a screenshot of the newly registered domain and determines if the site is merely parked or actively hosting a credential harvester. By the time the SOC analyst receives the alert, all the necessary evidence, including the screenshot, is already compiled. This allows analysts to immediately decide on action rather than spending time on initial investigation. The result is a paradigm shift: the security team identifies and reports malicious sites before the first victim ever encounters them, effectively pre-empting attacks.

Case 2: Automated Malware Triage

This demonstration showcased the pipeline's ability to automate the laborious process of malware analysis. The system monitors various malware sharing platforms and Telegram channels. When a new malware sample appears, dedicated bots automatically download it and submit it to a sandbox environment, such as Cuckoo or Triage. The sandbox generates a detailed report, from which the pipeline's parser automatically extracts all relevant Indicators of Compromise (IOCs), including C2 IP addresses and domains. These high-confidence IOCs are then immediately pushed to the SIEM watchlist and EDR blocklists. This automation has a staggering impact: the time an analyst previously spent on manually triaging a single malware sample (approximately 30 minutes) is reduced to zero. Analysts are no longer burdened with "file moving" tasks and only engage with samples flagged as truly noteworthy, freeing them to focus on high-value hunting and investigation.

Case 3: Closing the Loop (SOC to CTI)

This case study underscores the unique self-healing capability of the pipeline, distinguishing it from standard automation. The system isn't asking analysts to do more work; rather, it captures the inherent value of the work they are already doing. By simply recording the final decision on whether an alert represents a true incident or a false positive, analysts provide crucial feedback to the orchestration layer.

• If a source consistently feeds noisy data that leads to false positives, its trust score within the system gradually fades until the system automatically drops it, preventing future alert fatigue.
• Conversely, if an alert from a specific source leads to a confirmed incident, the system not only increases the trust associated with that source but also intelligently pivots to find related indicators from the same campaign, expanding the scope of detection.

This means the system evolves daily, with the SOC's ongoing investigation process serving as its continuous training data. There's no need for additional funds or manual updates; the intelligence grows organically with every analyst's click.

These case studies collectively demonstrate how the modular, automated CTI pipeline transforms security operations, dramatically improving speed, accuracy, and efficiency while empowering analysts to perform more strategic work.

Defensive Implications

▶ Watch: Q&A: Making CTI reports actionable and project build timeline (24:00)

The detailed architecture and proof-of-concept demonstrations presented by Daniel Lima and Gabriel Testoni offer profound implications for security defenders seeking to enhance their operational efficiency and effectiveness. The most critical takeaway is the necessity of shifting focus from mere data collection to actionable intelligence. Defenders must critically evaluate whether their CTI efforts are truly driving defensive measures or merely generating reports. If intelligence isn't leading to action, the approach needs to be rethought.

Defenders should prioritize automation for speed. In an environment where adversaries operate at machine speed, human-driven data movement is a critical liability. Implementing API-driven processes and scripts for collection, enrichment, and dissemination can drastically reduce the time to block threats, as demonstrated by the reduction from six hours to five minutes. This frees analysts from mundane tasks, allowing them to engage in higher-value activities such as threat hunting, complex incident response, and strategic analysis. The project achieved a significant reduction in false positive rates, from 30% to 3%, by implementing a trust scoring system and feedback loops, proving that automation can lead to greater accuracy, not just speed.

The adoption of a modular architecture (the "LEGO brick" approach) is crucial for resilience and adaptability. Security teams should design their CTI pipelines with independent components that can be updated or replaced without disrupting the entire system. This ensures continuous operation even as threat landscapes and data sources evolve. Furthermore, implementing a tiered response mechanism based on confidence levels (high, medium, strategic) allows for appropriate allocation of resources and speed of action. High-confidence threats can be automatically blocked, while medium-confidence alerts receive necessary human scrutiny, optimizing response capabilities.

Finally, defenders must learn to measure and report meaningful business impact. Instead of focusing on technical metrics or raw data volume, communication with leadership should emphasize analyst hours saved, reduced time to block, and improved false positive rates. This "language that bridges the gap between the SOC and the management" is essential for securing the necessary budget and resources to scale automation projects. For organizations just starting, the advice is to start small, identifying a single pain point like automating a phishing inbox, and leveraging readily available open-source tools such as Shovel, N8N, The Hive, and MISP, which prove that high-impact automation is accessible regardless of budget constraints. The ultimate defensive mindset should be to "don't react, anticipate," transforming knowledge into action faster than adversaries can adapt.

Key Takeaways

• Action is the Metric: Cyber Threat Intelligence (CTI) must directly drive defensive actions. If CTI efforts primarily produce reports without leading to concrete protection, the underlying approach needs fundamental re-evaluation.
• Automate for Speed: Adversaries operate at machine speed, and defensive measures must match this pace. Automate data collection, enrichment, and dissemination processes to drastically reduce time to block and free analysts for high-value tasks.
• Build a Feedback Loop: Implement a self-healing pipeline where SOC investigation outcomes (e.g., incident confirmation or false positive identification) feed directly back into the CTI orchestration. This allows the system to learn, improve trust scoring, and adapt automatically.
• Start Small, Measure Impact: Don't attempt to automate an entire workflow at once. Identify a single, high-pain point (like phishing inbox triage), automate it, and meticulously measure the analyst hours saved and other tangible business impacts to gain buy-in and resources for scaling.
• Leverage Open Source: High-impact automation does not require expensive commercial platforms. Robust open-source tools like Shovel, N8N, The Hive, and MISP can form the foundation of an effective and cost-efficient CTI pipeline.
• Anticipate, Don't React: The ultimate goal of an intelligent, automated CTI pipeline is to shift security operations from a reactive stance to a proactive one, enabling anticipation of threats before they impact victims.

About the Speaker(s)

Daniel Lima is a representative of NTT DATA, where he plays a significant role in managing security operations across Latin America. Specifically, Daniel oversees four Security Operations Centers (SOCs) located in Chile, Brazil, Peru, and Mexico, providing critical cybersecurity services to a diverse range of customers with varying contexts and maturity levels. His insights are grounded in real-world challenges and the practical application of advanced CTI solutions within a large service provider environment.

Gabriel Testoni is also from NTT DATA and brings over six years of specialized experience in the field of cyber threat intelligence. His expertise spans cybercrime investigations and risk intelligence. In his role, Gabriel focuses on correlating all SOC activities with CTI to transform the daily influx of billions of indicators into actionable intelligence for clients. He is a hands-on builder of the modular and automated CTI pipeline discussed in the talk, demonstrating practical solutions to operational challenges.