CTI Agent Automated Battlecards from CTI Reports

CTI Agent Automated Battlecards from CTI Reports

Speakers: Mohamed Nabeel, PhD, Cyber Security Veteran, Palo Alto Networks

Conference: Recon Village

YouTube: https://www.youtube.com/watch?v=Y8xMr2JwyjU

Overview

In the dynamic landscape of cyber security, staying ahead of sophisticated threat actors requires timely, accurate, and actionable intelligence. However, a significant challenge lies in the sheer volume and unstructured nature of Cyber Threat Intelligence (CTI) reports. These reports, often published in natural language across various platforms, are difficult to process at scale, contain implicit tactics, techniques, and procedures (TTPs), and frequently feature indicators of compromise (IOCs) that are outdated by the time of publication. Mohamed Nabeel, a cybersecurity veteran and PhD from Palo Alto Networks, presented an innovative solution at Recon Village, addressing these critical pain points.

Nabeel's talk, "CTI Agent Automated Battlecards from CTI Reports," introduces an agentic system powered by Large Language Models (LLMs) designed to automate the collection, synthesis, and structuring of CTI. The core output of this system is a "threat actor battlecard" – a comprehensive, structured summary of a specific threat actor's activities, including their TTPs, tools, campaigns, and associated IOCs. This automated approach aims to transform the consumption of CTI, making it more efficient, less error-prone, and significantly more actionable for defenders.

The significance of this work extends beyond mere summarization. By leveraging advanced machine learning techniques, including unsupervised learning for clustering and graph expansion for proactive IOC identification, the system not only structures existing intelligence but also actively discovers new, relevant indicators. This capability directly addresses the challenge of stale IOCs, providing defenders with a continuously updated and enriched view of threat actor infrastructure, thereby enhancing proactive detection and response capabilities.

Background

▶ Watch: Problem: Unstructured CTI reports and implicit TTPs (1:00)

The genesis of this research stems from the inherent difficulties associated with consuming and operationalizing public cyber threat intelligence. CTI reports, while invaluable, are predominantly written in natural language, making automated extraction of actionable insights exceptionally challenging. This problem is compounded by several factors: reports often include diverse modalities like images and videos, may contain conflicting or incomplete information, and critically, frequently present TTPs implicitly rather than explicitly. Analysts must infer these crucial details based on their domain knowledge, a process that is both time-consuming and susceptible to human error. Furthermore, the lifecycle of an IOC means that by the time a report is published and disseminated, many of the listed IOCs may already be defunct or rotated by the threat actor, limiting their defensive utility.

To tackle these issues, Nabeel and his team sought to develop an agentic solution capable of transforming ambiguous and vague threat reports into highly structured, consumable battlecards. The proposed system builds upon several established and emerging technologies. It integrates unsupervised learning for grouping related threat reports, harnesses the power of Large Language Models (LLMs) for natural language understanding and generation, and employs agentic systems – LLMs operating in a loop with access to tools and memory – for complex information extraction and synthesis. The system grounds its understanding of TTPs by referencing the widely adopted MITRE ATT&CK framework, which categorizes adversarial tactics, techniques, and procedures across 14 tactics and over 300 techniques.

The overarching goal is to move beyond manual, labor-intensive CTI analysis towards an automated pipeline that can provide structured, up-to-date intelligence. This structured output, the "battlecard," is designed to be programmatically consumable, facilitating faster integration into security operations centers (SOCs) and enabling more robust, automated enforcement of defensive measures. The work also acknowledges the "garbage in, garbage out" principle of machine learning, emphasizing the importance of starting with high-quality data from trusted sources to ensure the reliability of the generated intelligence.

Key Findings

▶ Watch: Garbage in, garbage out: Importance of data quality (3:50)

The research presented by Mohamed Nabeel yielded several significant findings that advance the state of automated CTI processing:

• Superiority of LLM-Generated Summary Embeddings for Clustering: A pivotal finding was that clustering threat reports based on LLM-generated summaries consistently outperformed direct document embedding. When LLMs were prompted to summarize threat actors from reports, the resulting summaries exhibited greater homogeneity, leading to more distinct and less overlapping clusters (as measured by higher silhouette scores). Specifically, Gemini embedding from large language models consistently outperformed traditional embeddings like BERT and its variants in this task. This approach also proved to be faster and more cost-effective.

• Effectiveness of Agentic Systems for Battlecard Generation: The talk underscored that agentic systems – LLMs operating in a loop with access to external tools and memory – are significantly more effective at extracting structured battlecards from raw CTI reports than simpler LLM prompting or Retrieval Augmented Generation (RAG) approaches. The React pattern (Reason, Act) was specifically highlighted as a successful planning mechanism, allowing the agent to observe, think, and decide which tools to use iteratively until a comprehensive battlecard is produced.

• Robust Battlecard Extraction via MapReduce-like Synthesis: To overcome the inherent token limits and processing complexities of LLMs when dealing with multiple, lengthy threat reports, the team developed a MapReduce-like approach. Individual agents were tasked with generating battlecards for single reports in parallel, followed by a dedicated synthesis LLM that combined these individual outputs into a single, comprehensive battlecard for the threat actor. Intriguingly, the synthesis LLM was observed to generate actual code to deterministically combine the information, demonstrating a sophisticated level of autonomous problem-solving.

• Proactive IOC Identification through Graph Expansion: One of the most impactful findings is the development of a graph expansion technique for automatically identifying new, previously unobserved IOCs associated with threat actors. By leveraging the common attacker behavior of rotating and reusing infrastructure (e.g., IPs, certificates, fishing kits), the system builds association networks around known IOCs. Through graph embedding, clustering, and attribution logic, it can discover new malicious domains and IPs attributed to specific threat actors. This capability provides a crucial, proactive layer of defense, as demonstrated by the identification of IOCs for groups like Lazarus, Revolver Rabbit, and Kimsuki that had low detection rates (e.g., VT 102) among security vendors at the time.

• Structured and Actionable CTI Output: The ultimate deliverable—the threat actor battlecard—is a highly structured report encompassing a threat actor summary, aliases, location, campaigns (with start/end dates, targeted industries), associated IOCs, TTPs (grounded in MITRE ATT&CK), tools used, malware binaries, exploited CVEs, and source reports. This structured format significantly reduces ambiguity and enables programmatic consumption, drastically improving the utility and operationalization of CTI for security teams.

Technical Deep Dive

▶ Watch: Structure of the automated threat actor battle card (4:15)

The system for generating automated CTI battlecards is architecturally designed as a four-step pipeline, leveraging a combination of data collection, unsupervised learning, LLM agentic systems, and graph-based intelligence expansion.

Overall Architecture

1. Report Collection: Gathering a diverse set of CTI reports.
2. Threat Actor Grouping: Clustering these reports by the threat actor they describe.
3. Battlecard Generation: Using an LLM agentic system to extract and synthesize structured battlecards from grouped reports.
4. IOC Update: Automatically enriching battlecards with new, up-to-date IOCs using graph expansion.

Data Collection

The foundation of any robust intelligence system is high-quality data. The system ingests reports from trusted sources including reputable security vendors, investigative journalists, and curated GitHub repositories and Twitter handles. This selective approach is crucial to mitigate the "garbage in, garbage out" problem, ensuring that the LLMs process reliable and relevant information. From these reports, the system aims to extract key structural entities: a summary of the report, the campaign it discusses, the specific threat actor, associated indicators, and TTPs. TTPs are rigorously grounded in the MITRE ATT&CK matrix, providing a standardized framework for classification.

Report Grouping (Clustering)

Once reports are collected, the next challenge is to group them by the relevant threat actor. This is achieved using unsupervised learning techniques, specifically clustering. Two main strategies were evaluated:

1. Document Embedding Clustering: Directly taking raw threat report documents, generating embeddings (numerical representations), and then clustering these embeddings.
2. Summary Embedding Clustering: Prompting an LLM to generate a concise summary of the threat actor from each report, then taking the embeddings of these summaries for clustering.

The research conclusively found that the summary embedding strategy significantly outperformed direct document embedding. The reason for this superiority lies in the LLM's ability to create more homogeneous summaries, consistently identifying the threat actor, their aliases, and core activities. When converted into embeddings, these consistent summaries result in vectors that are spatially closer for reports pertaining to the same threat actor, yielding better-defined clusters with less overlap. The Gemini embedding model from Google's LLMs demonstrated superior performance over other models like BERT and its derivatives, as evidenced by higher silhouette scores (a metric for cluster separation). This method also offered advantages in speed and cost efficiency.

Battlecard Generation (Agentic System)

The heart of the battlecard generation lies in an agentic system powered by LLMs. Unlike simple prompting or Retrieval Augmented Generation (RAG), agentic systems involve an LLM operating in a loop, equipped with access to various tools and potentially a memory to retain context. This "LLM in a for loop" paradigm allows for iterative problem-solving.

• Planning Patterns: The system employs the React pattern (Reason, Act), a common agentic planning approach. In React, the agent observes its environment (e.g., the CTI report), thinks about the next step, decides which tool to call, and then acts by invoking that tool. This cycle repeats, updating the environment with tool outputs, until a final answer (the battlecard) is achieved.

• Custom Tools: The effectiveness of an agentic system heavily relies on the quality and utility of its tools. Three custom tools were developed:
• Report Collection Tool: A simplistic tool to fetch and store necessary information from the threat reports locally, making it accessible to other tools.
• Search Tool: When the agent encounters ambiguous or missing information in a report, it can invoke this tool to query external sources for additional context.
• Extractor Tool: This is the core tool responsible for analyzing CTI reports and identifying the specific artifacts required for the battlecard (e.g., IOCs, TTPs, tools).

• Battlecard Extraction Strategy: An initial approach of feeding all grouped reports for a threat actor to a single LLM failed due to token limits and processing complexity. To address this, a MapReduce-like strategy was implemented:

1. Map Phase: Each individual threat report is processed in parallel by a dedicated agent. This agent generates a preliminary battlecard specific to that single report.
2. Reduce (Synthesis) Phase: Once individual battlecards are generated, a separate synthesis LLM takes all these battlecards and combines them into one comprehensive, consolidated battlecard for the entire threat actor. Notably, the synthesis LLM was observed to generate actual code (e.g., a Python script) to deterministically merge the information from multiple battlecards, demonstrating an advanced capability for structured output generation.

The resulting battlecard is a highly structured document, including:

• Threat actor summary, aliases, and known locations.
• Details of one or more campaigns, including start/end dates and targeted industries.
• Associated IOCs (IPs, domains, hashes).
• TTPs, explicitly grounded in the MITRE ATT&CK framework.
• Tools and malware binaries used.
• Exploited CVE vulnerabilities.
• Sources from which the information was derived.

IOC Update (Graph Expansion)

Recognizing that IOCs in published reports quickly become outdated, the system incorporates a graph expansion technique to proactively discover new, associated IOCs. This technique exploits the common attacker behavior of rotating and reusing their infrastructure, thereby creating detectable associations.

• Intuition: Attackers often use shared infrastructure (e.g., the same IP address hosting multiple malicious domains, or common certificate properties). By mapping these connections, new malicious assets can be identified and attributed. An example shown was the Kimsuki threat actor, which started with a few seed domains and then progressively created many more, all sharing infrastructure.
• Pipeline:

1. Seed IOCs: Start with known IOCs from the generated battlecard.
2. Association Network Building: Construct a network graph by identifying associations between these seed IOCs and other entities (e.g., IP addresses, SSL certificates, phishing kits, ASN information).
3. Graph Enrichment: Add properties to the nodes in the graph (e.g., IP reputation, URL characteristics, ASN details).
4. Graph Embedding: Generate graph embeddings that capture both the topological relationships and the properties of the nodes within the network.
5. Clustering: Cluster these graph embeddings to identify groups of related infrastructure.
6. Attribution and Update: Within each cluster, if a significant number of known threat actor-attributed domains exist, and other domains in that cluster are identified as malicious, then those newly identified malicious domains are attributed to the same threat actor. The battlecard is then updated with these fresh IOCs.

This sophisticated pipeline allows for the identification of new IOCs that are often unknown to the broader security community (e.g., having a VT 102 detection rate, meaning only 1 or 2 vendors detect them on VirusTotal), providing a significant proactive advantage.

Demo / Proof of Concept

▶ Watch: LLM-generated summary embedding outperforms direct document embedding (6:00)

While the presentation did not feature a live, interactive demonstration, Mohamed Nabeel provided compelling evidence and examples of the system's functionality and output.

The technical deep dive included a visual representation of an agent's execution trace, specifically for the threat actor Scattered Spider (also known as UNC3944 or Lapsus$ subgroup). This trace illustrated the React pattern in action, showing the agent iteratively calling tools (e.g., the extractor tool, search tool), observing the results, reasoning about the next step, and ultimately progressing towards the final battlecard. Each step in the loop, from initial observation to tool invocation and subsequent thought, was clearly delineated, demonstrating the systematic approach of the agentic system.

Furthermore, a complete example of a generated battlecard was presented for the threat actor "Model Libra" (an alias often associated with Scattered Spider). This battlecard showcased the structured format, including a detailed summary, identified campaigns, associated IOCs, a clear listing of TTPs grounded in MITRE ATT&CK, tools used, malware binaries, exploited CVEs, and the source reports from which the intelligence was derived. This example concretely demonstrated the system's ability to synthesize complex, unstructured data into a highly organized and actionable format.

Finally, the talk presented the real-world impact of the graph expansion technique by showing newly identified IOCs for prominent threat actors such as Lazarus Group, Revolver Rabbit, and Kimsuki. These examples highlighted domains and IP addresses that were created after the initial CTI reports were published, underscoring the system's ability to provide continuously updated intelligence. Crucially, Nabeel pointed out that many of these newly discovered IOCs had very low detection rates on platforms like VirusTotal (e.g., VT 102), indicating their novelty and the proactive advantage offered by the system's graph expansion capabilities. These "proof-of-concept" outputs effectively demonstrated the practical utility and advanced capabilities of the CTI agent system.

Defensive Implications

▶ Watch: Explanation for summary embedding's superior clustering performance (8:00)

The CTI Agent Automated Battlecards system offers profound implications for cybersecurity defenders, transforming how threat intelligence is consumed, operationalized, and leveraged for proactive defense.

• Automated and Structured CTI Consumption: The primary benefit is the ability to automatically convert verbose, unstructured CTI reports into highly structured, machine-readable battlecards. This eliminates the manual, time-consuming, and error-prone process of human analysts sifting through reports, allowing security operations centers (SOCs) to ingest and act on intelligence significantly faster. The standardized format ensures consistency and reduces ambiguity, making it easier to integrate into existing security tools and workflows.

• Proactive Threat Detection and Blocking: The graph expansion technique is a game-changer for proactive defense. By identifying new malicious infrastructure (domains, IPs) associated with known threat actors before they are widely recognized, defenders gain a critical lead time. This enables the implementation of proactive blocking measures at network perimeters, email gateways, and endpoint detection and response (EDR) systems, potentially preventing attacks before they can fully materialize. The identification of IOCs with low VirusTotal (VT) detection rates (VT 102) underscores this capability.

• Enhanced Situational Awareness and Context: Comprehensive battlecards provide a holistic view of threat actors, detailing their TTPs (grounded in MITRE ATT&CK), tools, malware, exploited CVEs, and campaign specifics. This rich context allows defenders to understand not just what happened, but how and why, facilitating more informed decision-making, better threat hunting strategies, and more accurate risk assessments.

• Resource Optimization: Automating the laborious task of CTI analysis frees up valuable human analyst time. Instead of spending hours on report parsing and summarization, analysts can focus on higher-value activities such as strategic threat hunting, incident response, and developing sophisticated detection rules, thereby optimizing security team resources.

• Improved Vulnerability Management: By explicitly listing CVE vulnerabilities exploited by specific threat actors within the battlecards, the system provides direct, actionable intelligence for vulnerability management teams. This allows organizations to prioritize patching and mitigation efforts based on the actual threats they face from known adversaries.

• Deterministic and Consistent Intelligence: The use of agentic systems and a MapReduce-like synthesis process, which can even generate code for deterministic combination, ensures a consistent and repeatable output. This reduces variability inherent in human analysis and provides a reliable source of truth for threat actor profiles.

In essence, this system empowers defenders to move from a reactive stance, constantly playing catch-up with threat actors, to a more proactive and predictive security posture, armed with timely, structured, and continuously updated intelligence.

Key Takeaways

• LLM-generated summaries significantly improve the accuracy and distinctness of threat actor clustering, outperforming direct document embeddings and leading to more effective organization of CTI reports.
• Agentic systems, specifically leveraging the React pattern with custom tools, are crucial for robust and comprehensive extraction of structured intelligence from natural language CTI reports, surpassing simpler LLM prompting or RAG methods.
• A MapReduce-like processing and synthesis approach effectively overcomes LLM token limits and complexity, enabling the creation of a single, consolidated battlecard from multiple individual threat reports.
• Graph expansion is a powerful technique for proactively identifying new, unobserved IOCs by leveraging attacker infrastructure rotation and reuse patterns, offering a critical advantage in early threat detection.
• Automated battlecard generation and continuous IOC updates provide timely, structured, and actionable intelligence, empowering defenders with enhanced situational awareness and enabling more proactive security measures.
• Future advancements include incorporating multimodal information (images, charts) from reports and further refining human-in-the-loop validation to enhance the agent's accuracy and reduce potential hallucinations.

About the Speaker(s)

Mohamed Nabeel is a distinguished figure in the field of cybersecurity, holding a PhD in cybernetics. As a seasoned cybersecurity veteran, he brings extensive experience to his work, currently leading efforts at Palo Alto Networks in proactive detection and graph-based threat intelligence research. Beyond his corporate role, Nabeel is a passionate advocate for open-source initiatives and is an active member of the Apache Software Foundation, contributing to the broader technology community. His expertise spans advanced machine learning applications in security, particularly in leveraging cutting-edge techniques like LLMs and graph theory to solve complex intelligence problems.