Beyond Manual Hunting: OSINT Copilot with LLMs & Evidence-Based Collection, Enrichment ...

Beyond Manual Hunting: OSINT Copilot with LLMs & Evidence-Based Collection, Enrichment ...

Speakers: Matheus Bezerra (Apura Cyber Intelligence S/A); Victor Fabre (Apura Cyber Intelligence)

Conference: FIRSTCON

YouTube: https://www.youtube.com/watch?v=3M5y8bwLZ9s

Overview

In an era defined by an ever-expanding digital attack surface and increasingly sophisticated threat actors, the traditional approach to cybersecurity threat hunting often falls short. Analysts are overwhelmed by vast quantities of unstructured data, struggling to sift through noise, correlate disparate pieces of information, and derive actionable intelligence. This talk, "Beyond Manual Hunting: OSINT Copilot with LLMs & Evidence-Based Collection, Enrichment...", presented by Victor Fabre and Matheus Bezerra of Apura Cyber Intelligence S/A at FIRSTCON, addresses these critical challenges head-on. The speakers introduce an innovative system that leverages Large Language Models (LLMs) and robust data engineering principles to transform raw, chaotic Open Source Intelligence (OSINT) into structured, searchable, and ultimately, actionable insights.

The core proposition is an OSINT Copilot designed to augment human analysts, moving beyond the limitations of manual hunting. By automating the collection, normalization, and enrichment of data from diverse sources like forums and marketplaces, the system aims to reduce the burden on highly skilled analysts. The integration of LLMs facilitates sophisticated semantic analysis, summarizing noisy events, identifying intent, and flagging risks, thereby enabling analysts to focus on high-priority threats and make informed decisions more rapidly. This approach represents a significant step forward in operationalizing OSINT for proactive threat intelligence and defense.

Background

▶ Watch: Introduction: hunting challenges and LLM solution (0:00)

The landscape of cybersecurity hunting is fraught with challenges that have historically constrained its effectiveness. Analysts often find themselves akin to "cavemen with a spear," manually tracking digital prey across a wilderness of data. The primary issues stem from the sheer volume and chaotic nature of OSINT sources, which include dark web forums, illicit marketplaces, social media, and open web intelligence. This deluge of information presents several hurdles:

1. Data Overload and Noise: Analysts must contend with immense amounts of data, much of which is irrelevant or "noise," making it difficult to pinpoint genuine threats or indicators of compromise (IOCs).
2. Skill and Experience Dependency: Effective hunting requires analysts to possess a high degree of skill, experience, and domain-specific knowledge to understand the context, nuances, and implications of the data they uncover. This expertise is a scarce resource.
3. Lack of Structure and Context: Much of the valuable OSINT is unstructured text, riddled with slang, abbreviations, and implicit meanings. Extracting meaningful context—such as the origin, time, author, and connections—is paramount, yet incredibly challenging at scale without automated tools. A simple post's text alone is insufficient; understanding the full thread, author's history, attachments, and categorization is vital for comprehensive analysis.
4. Reporting and Actionable Intelligence: Even when valuable data is found, organizing it into a coherent, actionable report for decision-makers is a time-consuming and labor-intensive process. Raw data must be transformed into intelligence, not just presented as an uncontextualized dump.

Prior work often relied on static rules, regular expressions, and keyword matching, which are easily bypassed by threat actors using varied language or new communication patterns. These methods struggle with the dynamic and often adversarial nature of OSINT, where context, intent, and subtle relationships are crucial. The need for a more dynamic, intelligent system that can understand and correlate information beyond simple pattern matching is evident, paving the way for AI-driven solutions to augment human capabilities in this complex domain.

Key Findings

▶ Watch: Semantic analysis using LLMs for data enrichment (6:00)

The talk highlights several key findings and contributions that collectively define a novel approach to OSINT-driven threat hunting:

1. The Primacy of Data Engineering: A fundamental finding is that LLMs are not a silver bullet; their effectiveness is directly proportional to the quality and structure of the data they process. The speakers emphasize that a robust data engineering pipeline—encompassing sophisticated collection, meticulous normalization, and comprehensive enrichment—is the foundational product. Without well-prepared, contextualized data, LLMs will merely process "garbage context" and yield poor results.
2. Small LLMs for Scalable, Cost-Effective Intelligence: A crucial discovery is that small LLMs running on-cluster can achieve comparable accuracy to mainstream, larger models like those from Anthropic or OpenAI for core tasks such as risk flagging and entity extraction. This finding is critical for organizations dealing with millions of events and billions of tokens daily, as it drastically reduces computational costs and ensures data privacy by keeping sensitive information within the organization's control. Benchmarks showed that while summaries might differ slightly, key outputs like risk flags and mentions remained consistent across model sizes.
3. Context-Rich, Evidence-Based Collection: The system demonstrates that collecting not just text but also its context (source, time, author, thread, attachments, connections) is paramount. This allows for the creation of a graph connection of events, enabling more meaningful semantic investigations and correlations across diverse sources.
4. Automated Semantic Analysis and Prioritization: Beyond deterministic IOC extraction, the integration of LLMs for semantic analysis is a significant finding. This enables the system to understand nuance, intent (e.g., "credential sale"), and risk flags (e.g., "SSO panel sale"), which static rules often miss. This capability, combined with monetization signals, allows for the automated prioritization of events, ensuring that analysts focus their efforts on high-priority, actionable intelligence rather than sifting through irrelevant data.
5. Graph-Based Deep Search for Relationship Discovery: The most impactful finding is the system's ability to perform deep searches and construct complex relationships from minimal input. By providing an LLM agent with access to the indexed, enriched data and tools to query it, the system can automatically identify groups, members, communication channels, operational patterns, and involved IOCs, even when starting with just a single name or identifier. This automated correlation capability significantly accelerates the understanding of threat actor networks.

These findings collectively present a paradigm shift, moving from reactive, manual data sifting to proactive, AI-augmented intelligence generation, making OSINT a more scalable and effective component of a comprehensive cybersecurity strategy.

Technical Deep Dive

▶ Watch: Real-world example: transforming raw data into intelligence (11:30)

The architecture proposed by Victor Fabre and Matheus Bezerra is a sophisticated, multi-stage pipeline designed to transform raw OSINT into actionable intelligence. This process is highly dependent on robust data engineering at every step.

Data Collection: Bridging the Gap Between Lightweight and Heavyweight Scraping

The initial phase involves collecting data from a myriad of chaotic sources, including forums, marketplaces, and chat platforms. The speakers outline two primary methods, each with its own trade-offs:

1. Lightweight Extraction: This method utilizes standard HTTP clients to make requests and retrieve structured data (e.g., JSON) or static HTML.

• Pros: Requires fewer resources, easier to scale.
• Cons: Cannot execute JavaScript, making it ineffective for dynamic websites. Easily blocked by anti-bot gateways. Limited control over the fingerprint (the unique identifier connecting a device and network). Limited interaction with the rendered Document Object Model (DOM).

1. Browser Automation: When lightweight methods fail due to blocks or dynamic content, browser automation becomes necessary. This involves controlling headless browsers (e.g., via tools like Playwright or Puppeteer, though not explicitly named, the concept is the same).

• Pros: Can execute JavaScript, interact with forms, bypass complex anti-bot gateways, and create realistic browser fingerprints.
• Cons: Significantly higher resource consumption due to running full browser instances. Introduces more latency due to increased requests and processing. Dramatically increases complexity, especially when needing to run hundreds or thousands of browsers concurrently.

A robust collection pipeline must address several challenges: dynamically changing fingerprints, bypassing anti-bot measures, solving CAPTCHAs, adhering to rate limits, and effectively managing sessions and cookies to avoid detection as a bot and maintain persistent logins.

Data Normalization and Enrichment: From Raw Text to Semantic Understanding

Once data is collected, it undergoes a critical transformation process:

1. Normalization: This involves standardizing the collected data into a consistent event schema. Key steps include deduplication to remove redundant information and language detection to categorize content for subsequent processing.
2. Deterministic Analysis: This layer extracts specific, well-defined Indicators of Compromise (IOCs) using regular expressions and validators. Examples include email addresses, IP addresses, URLs, domains, and cryptocurrency wallet addresses. A content hash is often computed for these extracted entities to facilitate easy comparison and correlation across events.
3. Semantic Analysis with LLMs: This is where the power of AI truly enhances the pipeline. Deterministic methods struggle with slangs, abbreviations, and nuanced language that are common in OSINT. LLMs are employed to perform semantic analysis, extracting deeper meaning and context:

• Dimension: Identifying if a specific entity (e.g., a company) is being discussed.
• Intent: Determining the purpose of the conversation (e.g., "credential sale").
• Contact: Understanding how individuals are interacting.
• Risk Flags: Automatically identifying specific threats (e.g., "SSO panel sale" indicates "credential abuse").
• Summary Generation: LLMs create concise summaries of potentially noisy or lengthy events, providing quick context for analysts.

1. Embeddings for Natural Language Search: From the extracted entities and LLM-generated summaries, the system creates search vectors (embeddings). These are multi-dimensional numerical representations of text that capture semantic meaning. By comparing the similarity of these vectors, the system enables natural language search, allowing users to query the data with context rather than just keywords. It's crucial to store these embeddings in a separate index for efficient searching.

LLM Selection and Operationalization: Small Models, Big Impact

A significant operational decision is the choice of LLMs. For processing millions of events and billions of tokens daily, using mainstream commercial models (like Anthropic or OpenAI) becomes prohibitively expensive. The speakers advocate for the use of small LLMs running within the organization's own cluster.

• Tools for Efficiency: They mention using tools like vLLM for efficient GPU KV cache management and MTP (multi-token prediction) to accelerate response times.
• Benefits of On-Cluster LLMs:
• Fixed Control: Complete control over the model's behavior and environment.
• Data Privacy: Sensitive data remains within the organization's infrastructure, avoiding third-party API exposure.
• Predictable Cost: While subject to some fluctuation, the average operational cost is more predictable than API-based pricing.
• Output Constancy: The output remains consistent, as the organization controls model updates and changes, unlike external APIs that may evolve.

Structured Data and Indexing

The culmination of the collection and enrichment phases is a highly structured event. Each event includes common fields (ID, author, topic, text, language), source-specific fields (forum name, thread ID, post ID), enrichment fields (indicators, mentions, risk flags), and temporal fields (created\_at for when the post was made, collected\_at for when it was scraped).

This structured data is then indexed using a distributed document search engine. The speakers specifically mention Elasticsearch as their chosen platform. Elasticsearch provides:

• Time-based indices: Efficiently managing and querying data over time.
• Embedding retrieval (with vector search capabilities) and BM25: Supporting both natural language (vector-based) and keyword-based search.
• Aggregations: Enabling powerful analytics by grouping data by authors, sources, mentions, and other criteria.

This indexing allows for flexible queries, from specific keywords and IOCs to complex natural language questions that leverage the semantic embeddings.

Demo / Proof of Concept

▶ Watch: Deep search: LLM agent uncovers threat actor networks (16:00)

The speakers illustrate the system's capabilities through concrete examples, demonstrating how it transforms raw data into actionable intelligence and facilitates deep investigations.

From Raw Text to Structured Intelligence: The Receita Federal Example

The first example shows a raw text snippet from a forum post discussing "Receita Federal" (Brazil's Federal Revenue Service), mentioning a "Buddha Group" and the sale of data. This unstructured, hard-to-read text is processed by the pipeline and transformed into a rich, structured event.

The resulting structured data includes:

• Metadata: Forum name, post user, topic ID, title, URL.
• Enrichment:
• Organization Mention: Explicitly identifies "Receita Federal organization."
• Summary: A concise LLM-generated summary of the event's content.
• Risk Flags: Automatically identifies potential risks related to data sale.
• Monetization Signals: Indicates that money or a price is involved, signaling a potential illicit transaction.

These last four fields (mentions, summary, risk flags, monetization signals) are crucial for prioritizing events. By focusing on events with high risk flags and monetization signals, the system ensures that analysts are not overwhelmed by the full volume of events but instead directed to the most critical intelligence, avoiding "garbage context" for further LLM analysis.

Routine Investigation: Bootstrap and Personalized Monitoring

The system supports two modes of continuous monitoring:

1. Bootstrap: For a broad, proactive approach, defenders provide basic information: company name, sector, and known nicknames. An LLM then leverages this context to generate a comprehensive set of keywords and queries optimized for searching relevant OSINT sources. This creates a continuous monitoring stream, generating alerts for new, high-priority events.
2. Personalized: For more targeted intelligence, users can define specific parameters such as domains, campaigns, or specific sources. This allows organizations (e.g., a bank with a new brand) to tailor their monitoring to highly specific threats, with LLMs generating precise queries for these custom requirements.

Both methods ensure continuous, automated surveillance of OSINT, providing timely alerts on emerging threats without relying on static, easily outdated search queries.

Deep Search: Contextual Navigation and Relationship Discovery

Perhaps the most compelling demonstration is the deep search functionality, which moves beyond simple event retrieval to reconstruct complex relationships. The problem statement is clear: searching for an author like "Gordon Freeman" might return 700 events. Manually reading and correlating these events to understand who is talking to whom and the broader context is an insurmountable task.

The solution involves an LLM agent equipped with tools to access the database and perform aggregations. Given just the name "Gordon Freeman," the agent performs the following:

• Event Analysis: It processes all 700 events associated with "Gordon Freeman."
• Relationship Discovery: It identifies a malicious group called "Latin F," active in countries like Venezuela, Argentina, Guatemala, Chile, Ecuador, and Canada.
• Actor Identification: It discovers other members of this crew, such as "Cypher for hunting tree," and contacts from other sources like "Home Hood one," who are interacting with "Gordon Freeman." It also identifies potential buyers of stolen credentials.
• Operational Patterns: The agent outlines the common patterns of operation and how interactions occur among these actors.
• Communication Channels: It discovers the communication channel used, specifically Signal.me.
• IOC Extraction: Because the underlying pipeline has already performed deterministic analysis, the agent can immediately provide the IP addresses, emails, and domains involved in these activities without requiring further manual research.

This deep search capability transforms a massive volume of disparate events into a cohesive, contextualized graph of threat actor activity, providing an unparalleled understanding of their network, tactics, and associated IOCs from a single starting point.

Defensive Implications

▶ Watch: Q&A: Good context is vital for LLM performance (23:00)

The OSINT Copilot system presented by Victor Fabre and Matheus Bezerra offers profound defensive implications for cybersecurity teams, enabling a more proactive, efficient, and intelligent approach to threat intelligence and incident response.

1. Proactive Threat Hunting and Early Warning: By continuously monitoring diverse OSINT sources and automatically flagging high-priority events, defenders can gain early visibility into potential threats. This includes detecting discussions about data breaches, credential sales, or planned attacks targeting their organization or sector before they materialize into full-blown incidents. The ability to prioritize events based on risk flags and monetization signals ensures that security teams focus on the most impactful threats.
2. Enhanced Incident Response: When an incident occurs, the system can rapidly contextualize known IOCs or actor names. Instead of manual correlation across disparate data sources, an LLM agent can quickly map out actor networks, identify associated campaigns, communication channels, and other relevant IOCs. This drastically reduces the mean time to respond (MTTR) by providing comprehensive intelligence at speed.
3. Strategic Intelligence Gathering: The ability to conduct "deep searches" and discover relationships between threat actors, groups, and their operational patterns empowers security teams to build richer threat profiles. This strategic intelligence can inform long-term defensive strategies, improve vulnerability management by anticipating attack vectors, and enhance security awareness training.
4. Optimized Resource Allocation: Automating the laborious tasks of data collection, normalization, and initial analysis frees up highly skilled human analysts to focus on complex problem-solving, strategic analysis, and decision-making. This maximizes the value of scarce human expertise, ensuring that resources are allocated where they can have the greatest impact.
5. Data-Driven Decision Making: The structured and enriched data, along with the ability to perform aggregations, provides a solid foundation for data-driven decision-making. Security leaders can use these insights to understand the most prevalent threats, assess their organization's exposure, and justify investments in specific security controls.
6. Improved Data Privacy and Cost Efficiency: By demonstrating the effectiveness of small, on-cluster LLMs, the system provides a blueprint for organizations to leverage advanced AI capabilities without compromising data privacy or incurring exorbitant costs associated with large commercial models. This is particularly relevant for sectors handling highly sensitive information.

In essence, the OSINT Copilot transforms OSINT from a chaotic data swamp into a well-organized, intelligent repository, providing defenders with the tools to stay ahead of adversaries in an increasingly complex threat landscape.

Key Takeaways

• Data Engineering is Paramount: A robust pipeline for collection, normalization, and enrichment is the absolute foundation for effective OSINT analysis. LLMs are powerful, but only when fed clean, structured, and contextualized data.
• Small LLMs Offer Scalable & Secure Solutions: On-cluster, smaller LLMs can achieve comparable accuracy to larger commercial models for key tasks like risk flagging and entity extraction, providing significant cost savings and crucial data privacy benefits for high-volume processing.
• Context Trumps Raw Text: Understanding the full context of OSINT events (author, time, source, thread, attachments) is critical for meaningful analysis and correlation, moving beyond simple keyword matching.
• Semantic Analysis Elevates Intelligence: LLM-driven semantic analysis goes beyond deterministic IOC extraction, identifying nuanced intent, risk flags, and generating summaries that provide deeper, actionable insights from unstructured data.
• Graph-Based Deep Search Uncovers Hidden Relationships: Automated agents leveraging enriched data and graph connections can rapidly identify complex threat actor networks, communication patterns, and associated IOCs from minimal starting points, a capability far beyond manual hunting.
• Prioritization Drives Efficiency: Implementing risk flags and monetization signals allows for automated prioritization of OSINT events, directing human analysts to the most critical threats and avoiding "garbage context" for further analysis.

About the Speaker(s)

Matheus Bezerra and Victor Fabre are cybersecurity professionals from Apura Cyber Intelligence S/A. Their presentation at FIRSTCON highlighted their expertise at the intersection of threat hunting and data engineering. Victor Fabre introduced the overarching problem of manual hunting and the vision for an LLM-powered copilot, while Matheus Bezerra delved into the technical intricacies of the data pipeline, LLM implementation, and the practical demonstrations. Their work at Apura Cyber Intelligence S/A focuses on developing innovative solutions to transform chaotic OSINT into actionable threat intelligence for defenders.