Phishing for Patterns: What Happens When Agents Explore Domain Data

Phishing for Patterns: What Happens When Agents Explore Domain Data

Speakers: Patrick Schläpfer, Beni Urech

Conference: Area41

YouTube: https://www.youtube.com/watch?v=JwnR_bD1g0o

Overview

In an era where phishing remains a pervasive and evolving threat, the ability to rapidly identify and track entire malicious campaigns is paramount for effective defense. This talk by Patrick Schläpfer and Beni Urech from Area41 explores a novel approach: leveraging AI agents with access to vast repositories of domain data to automatically uncover phishing campaigns. The presenters delve into the journey of evolving a specialized threat intelligence platform, born from a personal hobby, into a sophisticated system capable of interacting with Large Language Models (LLMs) to perform complex, generic threat hunts.

The core of their presentation lies in demonstrating how LLM-powered agents, equipped with custom tools and meticulously crafted prompts, can analyze billions of domain records, certificate transparency logs, and DNS information to detect the intricate patterns indicative of large-scale phishing operations. This work addresses a critical challenge in cybersecurity: moving beyond static, signature-based detections to dynamic, AI-driven analysis that can adapt to new threat actor methodologies. By showcasing the technical architecture, iterative development process, and comparative performance of various LLMs, Schläpfer and Urech provide invaluable insights into the future of automated threat intelligence and proactive defense strategies.

Background

▶ Watch: Introduction: AI agents detecting phishing campaigns (0:00)

The genesis of this ambitious project stems from Patrick Schläpfer's personal fascination with the threat actor TA505, known for campaigns like the STB bot and Get-go Downloader approximately two years prior to the talk. What captivated Schläpfer was TA505's highly recognizable domain registration patterns: consistently registering two new domains every day, often with similar keywords, specific registrars, and DNS service providers, predominantly using .com TLDs. This led him to develop a simple Python script utilizing Certificate Transparency logs and zone files to identify these domains early, aiming to block them within his network. While effective for TA505, this approach quickly revealed a significant limitation: adapting to new threat actors, such as TA551, with different patterns required substantial script rewrites, making it an unsustainable and "quite annoying" manual effort.

Recognizing the need for a more generic and adaptable solution, Schläpfer and Beni Urech embarked on a five-year journey to build a comprehensive domain intelligence system, which they affectionately call Shadow Spotter. This platform was designed to collect, store, and query vast amounts of domain information efficiently. Their architecture involved ingesting Certificate Transparency logs and zone files into Kafka, then using Vector to transform these messages and store them in a ClickHouse database. To enrich this raw data, they implemented active lookups for WHOIS records, DNS information, HTTP data, and JA3 fingerprints. The system was further empowered by a custom query language that allowed for flexible detection rule creation, adaptable to any desired threat actor. At the time of the talk, Shadow Spotter housed approximately 3.1 billion domains and 50 terabytes of queryable data, running at a personal hobby cost of around 450 euros per month.

The sheer scale and utility of Shadow Spotter presented a new challenge: how to make this powerful tool accessible and valuable to others. As engineers, the speakers admitted to not being "really good in talking to people" to acquire customers. This predicament, combined with the rise of AI, sparked a "brilliant idea": to "sell it to agents." This marked the pivotal shift from a human-centric UI to an agent-driven interaction model, necessitating a complete re-evaluation of how their rich dataset could be leveraged by autonomous AI systems.

Key Findings

▶ Watch: Understanding agent architecture, tools, and MCPs (4:30)

The talk revealed several critical insights into building and operating AI agents for advanced threat hunting:

• Agent Architecture and the Role of Tools: An AI agent fundamentally consists of a system prompt (defining roles and goals), tools (the interface between the LLM and external systems), skills (text files providing extended manuals for existing tools), and the LLM itself (which is stateless and requires the entire conversational history/context with each request). The agent loop orchestrates the interaction between these components.
• The Meta-Agent Communication Protocol (MCP): This standardized protocol proved crucial for enabling remote tools and prompts. MCP allows tools to be local (via standard input/output) or remote (via HTTP), making them interchangeable across different LLMs and agent frameworks (e.g., Claude, CodeX, Open Web UI). This standardization significantly simplifies tool integration and reusability.
• Iterative Tool Development: The most effective way to build powerful agent tools was through an iterative process. Starting with a REST API, they used Claude to translate it into an MCP server, effectively creating one tool per REST endpoint. The LLM then performed hunts, reasoned about errors or improvements, and provided feedback, which was used to refine the tools and backend. This "hunt-reason-apply-rerun" loop proved highly efficient.
• Cruciality of Tool Descriptions: For LLMs to effectively utilize tools, their descriptions must be exceptionally detailed and clear. This was particularly evident when dealing with their custom "Shadow Lang" query language; verbose instructions with examples were necessary to minimize "wrong tool errors," especially for local models. Tool descriptions act as the "user interface for the model."
• Output Format Matters for Token Efficiency: While REST APIs typically return JSON, the speakers found that CSV was a far more token-efficient format for tabular data. LLMs process both equally well, but JSON's verbosity wastes valuable tokens, increasing costs and potentially slowing down interactions.
• Hardcoding vs. Reasoning: LLM reasoning is expensive and prone to errors. For procedures that should always follow the same order or involve clear decisions, hardcoding them into compound tools (which wrap multiple API calls) is superior. The models, even advanced ones like Opus, sometimes made non-optimal choices, such as preferring JSON over CSV, suggesting that clear instructions or hardcoded paths are necessary where efficiency is critical.
• Agent Load on Backend Systems: AI agents, by their nature, can generate significantly higher load on backend systems than human users. Their ability to execute many calls in parallel can bring databases "to a halt," a critical consideration for infrastructure planning.
• Comparative Model Performance:
• Opus 48 and GPT 5.5 emerged as the clear leaders, with Opus excelling in the breadth of campaign discovery and GPT in the depth of analysis (e.g., extracting Telegram tokens).
• Qwen 36 (dense model) was a surprising strong performer among local models, often "punching above its weight" and finding unique clusters.
• Weaker models (Mimo, DeepSeek V4, Khemi, Qwen 36 MoE, Llama 4 dense/MoE) struggled with noise discipline (e.g., falsely assuming all domains on a Cloudflare IP are phishing), lacked depth of analysis, and provided poor verdicts (e.g., classifying dormant domains as 90% fishing).
• Reproducibility: While smaller campaign clusters were generally reproducible across models, larger or more elusive campaigns showed less consistency, with models identifying different facets or "colors" of the overall campaign.
• A Hybrid Approach: The optimal strategy involves combining the strengths of different models: using a breadth-focused model like Opus for initial campaign discovery, then a depth-focused model like GPT for detailed analysis and false positive identification. For local deployment, Qwen 36 (dense) serves well for triage, but requires human verification.
• The Dynamic AI Landscape: The rapid introduction of new LLMs means the landscape is constantly shifting, necessitating continuous curiosity and adaptation from security practitioners.

Technical Deep Dive

▶ Watch: Key learnings for improving tools and query languages (14:00)

The technical foundation of this work is the Shadow Spotter domain intelligence platform, meticulously crafted over five years. This system is designed for high-volume data ingestion and efficient querying. Data collection begins with ubiquitous sources like Certificate Transparency (CT) logs and zone files, providing a near real-time stream of newly registered or updated domains. This raw data is then fed into a robust ingestion pipeline: messages are first routed through Kafka, a distributed streaming platform, ensuring reliable and scalable data transport. From Kafka, Vector is used to transform the messages into a suitable format before being stored in ClickHouse, a high-performance columnar database optimized for analytical queries on large datasets.

Beyond passive collection, Shadow Spotter actively enriches its data. This includes performing WHOIS requests to gather registration details, DNS lookups for associated records, capturing HTTP information (e.g., status codes, redirects), and generating JA3 fingerprints for TLS client identification. This comprehensive dataset is made queryable through a custom query language, dubbed "Shadow Lang," which allows analysts to define complex patterns for threat detection, moving beyond the limitations of fixed scripts.

The integration of AI agents with Shadow Spotter necessitated a careful architectural design. An agent, in this context, is structured around several key components:

1. System Prompt: This establishes the agent's persona, its overarching goals, and operational policies.
2. Tools: These are the critical interfaces that allow the LLM to interact with external systems. Each tool is defined by a unique name, a detailed description (crucial for the LLM's understanding), and a fixed input schema for argument passing.
3. Skills: These are essentially text files that provide extended manuals or usage examples for existing tools, particularly useful for powerful but generic tools like bash or curl.
4. Large Language Model (LLM): This is the "intelligence" core. Crucially, LLMs are stateless; every request must include the entire conversational history and context for the model to maintain coherence.
5. Agent Loop: This orchestrates the entire process, managing prompts, tool calls, and LLM API interactions in a continuous cycle.

The communication between the LLM and the Shadow Spotter backend is facilitated by the Meta-Agent Communication Protocol (MCP). MCP provides a standardized way for agents to use remote tools and prompts, whether those tools are running locally (communicating via stdin/stdout) or on a remote server (via HTTP). This allows for flexibility, enabling various agent frameworks (e.g., Claude Code, CodeX, Open Web UI) to interact with the same backend services.

A concrete example illustrates the MCP workflow during a phishing hunt:

1. The agent sends the system prompt, a specific user message (e.g., "hunt for postfinance.ch-notfallhilfe.com in the last 30 days using Shadow Spotter MCP"), and a list of all available local tools (each with its name, description, and input schema) to the LLM.
2. The LLM processes this context and responds with a tool_use message, instructing the agent to invoke a specific tool, for instance, MCP Shadow Spotter Enrich Indicator, along with its required input parameters.
3. The agent validates these parameters against the tool's schema, confirms the tool's existence, and then calls the tool.
4. The tool, via the MCP server, translates this call into a request to the corresponding backend function within the Shadow Spotter system.
5. Shadow Spotter executes the function (e.g., retrieving enrichment information like domain creation date and registrar) and returns the results to the agent.
6. The agent appends these results to the existing context and sends the entire updated context back to the LLM, initiating the next iteration of the loop. This cycle continues until the LLM determines it has sufficient data to form a conclusion and generate a report.

The development of these tools was an iterative process, starting by translating existing REST API endpoints into MCP tools. The LLM (specifically Claude/Opus) was then used in a feedback loop: it would perform a hunt, reason about what worked or didn't, suggest improvements (e.g., a new tool for Cloudflare name server pairs, or better descriptions for graph tools), and then the developers would apply these changes, rerun the hunt, and compare results. This continuous refinement was crucial for optimizing tool effectiveness and LLM interaction.

Key technical optimizations discovered during this process include:

• Output Format: Switching from verbose JSON to dense CSV for tabular data significantly reduced token consumption, making interactions more efficient and cost-effective.
• Compound Tools: For repetitive or deterministic procedures, creating compound tools that hardcode a sequence of API calls or decisions prevented the LLM from expending tokens on reasoning where it wasn't needed, improving reliability and efficiency.
• Agent Control: The agent itself executes the tools, maintaining control over the interaction with external systems, rather than the LLM directly manipulating data.
• Backend Load Management: Recognizing that agents can generate far more concurrent requests than human users, the backend infrastructure of Shadow Spotter needed to be robust enough to handle high parallel loads.

Demo / Proof of Concept

▶ Watch: Comparing LLM performance in phishing campaign detection (20:00)

The core of the demonstration involved a series of experiments designed to evaluate the agents' efficacy in hunting phishing campaigns. The methodology was straightforward:

1. Manually identify an active phishing campaign and select one live domain as the starting point.
2. Feed this initial domain to the AI agent, instructing it to "search from this one domain the whole campaign of the fishing that is ongoing."
3. Two distinct playbooks were used: a strict playbook (a recipe-like, step-by-step guide) and a freestyle playbook (less guidance, allowing the LLM more creative freedom) to assess the impact of instructional rigor.
4. The desired outputs were a readable report for human analysis and a comprehensive list of Indicators of Compromise (IOCs) for consumption by other systems.

A wide array of LLMs were tested, categorized into three groups:

• Commercial Cloud Models: Opus 47/48 and GPT 5.5, consistently run on their highest settings.
• Chinese Cloud Models: Mimo, DeepSeek, and Kimiko, accessed via Open Code Go, also on highest settings.
• Local Models: Qwen (dense and Mixture of Experts - MoE) and Llama 4 (dense and MoE), deployed locally. These were run on Strix Halo devices with AMD Ryzen CPUs and 128 GB of RAM (leveraging unified memory technology) using a Llama server, with 8-bit quantized models for better performance.

Various coding agents were employed for these models: Cloud Code for Opus and the local models, CodeX for ChatGPT, and Open Code for the Chinese models. An initial custom agent was quickly abandoned due to its inferior performance compared to the readily available coding agents.

One compelling example of a successful hunt involved a campaign impersonating a local bank (SAKB). The agent not only identified the banking phishing but accurately determined it was specifically targeting Twint, a mobile payment service. The agent's report detailed how the phishing pages dynamically presented different UIs based on the user's selected bank. It successfully clustered domains related to Twint and even found related clusters around "Six" (another financial institution).

The comparative analysis of model performance yielded clear distinctions:

• Opus 48 and GPT 5.5 were the undisputed winners. Opus excelled in the breadth of its findings, uncovering a wide variety of campaign elements. GPT, conversely, demonstrated exceptional depth, performing in-depth content analysis, even extracting specific artifacts like Telegram tokens and IDs directly from the phishing pages.
• Qwen 36 (dense model) was a "really strong" and surprising local performer, often "punching above its weight" by identifying interesting and sometimes unique clusters. However, it still required human verification.
• Mimo was deemed "okay-ish," lacking the breadth and depth of the top models.
• DeepSeek V4, Khemi, Qwen 36 (MoE), and Llama 4 (dense and MoE) were the weakest performers. They frequently suffered from noise discipline issues, making incorrect assumptions such as "all domains on a Cloudflare IP are probably fishing," leading to high false positive rates. They also struggled with superficial content analysis (e.g., treating parked websites as phishing) and provided poor verdicts (e.g., classifying 90% of dormant domains as malicious).

Factors separating strong from weak models included:

• Noise Discipline: The ability to avoid simplistic, incorrect assumptions. The talk highlighted the effectiveness of pivoting on Cloudflare name server combinations and useful content, rather than just shared IPs.
• Depth of Analysis: Strong models could dissect phishing pages for actionable intelligence, while weak ones provided shallow, often misleading, analyses.
• Verdict Quality: Accurate and nuanced classification (unknown, fishing, suspicious) from strong models contrasted sharply with the over-aggressive and often wrong verdicts from weaker models.

Regarding reproducibility, smaller campaign clusters were more consistently identified across different models. However, for larger or more elusive campaigns, reproducibility decreased, with models tending to find different facets of the overall campaign.

The ultimate conclusion from the demo was a proposed hybrid strategy: use Opus for its broad campaign discovery, then leverage GPT for its deep, in-depth analysis and false positive filtering. For local, cost-effective triage, Qwen 36 (dense) was recommended, albeit with the caveat that human analysis is still necessary before taking automated actions like takedown requests or network blocks.

Defensive Implications

▶ Watch: Audience Q&A: Validating findings and false positive rates (28:50)

The findings presented in "Phishing for Patterns" offer several critical implications for cybersecurity defenders looking to enhance their capabilities against phishing and other domain-based threats:

• Automate Proactive Threat Hunting: The primary implication is the potential to automate the discovery of entire phishing campaigns from a single initial indicator. By feeding one known malicious domain to an AI agent, defenders can uncover related infrastructure and patterns that would otherwise require significant manual effort, dramatically increasing the speed and scale of threat intelligence.
• Adopt a Multi-Model AI Strategy: No single LLM is perfect. Defenders should consider a tiered approach, leveraging the strengths of different models. For instance, using models strong in breadth (like Opus) for initial campaign discovery and then feeding those results to models strong in depth (like GPT) for detailed content analysis (e.g., extracting C2 indicators, Telegram tokens) can provide a more comprehensive and accurate picture of a threat.
• Invest in Robust Tooling and MCP Integration: The success hinges on well-defined tools that allow LLMs to interact effectively with backend data sources. Adopting a standardized protocol like MCP (Meta-Agent Communication Protocol) can ensure tool reusability across various agent frameworks and LLMs, future-proofing the investment in tool development.
• Prioritize Clear Tool Descriptions and Examples: When building tools for LLMs, the quality of descriptions is paramount. Defenders should dedicate resources to crafting highly detailed instructions and providing concrete examples, especially for custom query languages or complex operations, to minimize LLM errors and optimize performance.
• Optimize for Token Efficiency: Be mindful of token usage. Switching from verbose JSON to denser formats like CSV for tabular data can significantly reduce costs and potentially speed up LLM interactions. Similarly, creating "compound tools" for repetitive or deterministic tasks can prevent LLMs from wasting tokens on unnecessary reasoning.
• Be Wary of LLM Reasoning Pitfalls: While powerful, LLMs can make simplistic assumptions leading to high false positive rates (e.g., attributing maliciousness to all domains sharing a Cloudflare IP). Defenders must build in validation steps and understand the limitations of the models they employ, especially for weaker or local models.
• Leverage Unique Pivoting Points: The talk highlighted effective pivoting points beyond simple IPs, such as Cloudflare name server combinations and content hashes/titles. Defenders should integrate these more sophisticated pivots into their hunting strategies to improve accuracy and reduce noise.
• Plan for Increased Backend Load: AI agents can generate a significantly higher volume of parallel requests than human users. Infrastructure supporting AI-driven threat hunting must be designed for scalability and resilience to handle this increased load without performance degradation.
• Embrace Continuous Learning and Adaptation: The AI landscape is evolving rapidly. Defenders must cultivate a "hacker mindset" to stay curious, experiment with new LLMs, and adapt their strategies to leverage emerging capabilities effectively. What works today might be superseded by a new model or technique tomorrow.
• Maintain Human Oversight for Critical Actions: While AI agents can automate discovery and analysis, human analysts remain crucial for verifying findings, especially before initiating high-impact actions like takedown requests or network blocks, particularly when using less robust local models for triage.

Key Takeaways

• Iterative Tool Development is Key: The most effective approach to building powerful AI agent capabilities is an iterative process where the agent itself helps create, use, criticize, and improve its own tools, leading to highly optimized and reliable interactions with backend systems.
• Model Selection Matters Immensely: Frontier models like Opus and GPT offer superior performance in terms of campaign breadth and analytical depth, respectively. While some local models (e.g., Qwen 36 dense) show promise for triage, weaker models are prone to high false positives and superficial analysis, underscoring the need to choose models appropriate for the task.
• Optimize LLM Interaction for Efficiency: To manage costs and improve performance, prioritize dense output formats like CSV over verbose JSON for tabular data, and hardcode common, deterministic procedures into "compound tools" to reduce reliance on expensive LLM reasoning.
• AI Agents Can Automate Complex Threat Hunting: LLM-powered agents, when integrated with rich domain intelligence platforms, can effectively discover and cluster entire phishing campaigns from a single indicator, significantly reducing manual effort and accelerating threat response.
• The AI Landscape is Rapidly Evolving: The constant emergence of new LLMs means defenders must maintain a "hacker mindset" – staying curious, continuously experimenting with new models, and adapting their strategies to leverage the latest advancements in AI for cybersecurity.
• A Multi-Model Approach Offers Best Results: Combining models strong in different areas (e.g., Opus for breadth of discovery and GPT for depth of analysis) allows for a comprehensive and robust threat hunting strategy, maximizing both coverage and accuracy in identifying phishing campaigns.

About the Speaker(s)

Patrick Schläpfer is a cybersecurity enthusiast whose journey into automated threat hunting began as a hobby, driven by his fascination with the domain patterns of threat actors like TA505. This personal interest evolved into the foundational work for Shadow Spotter, a large-scale domain intelligence platform designed for generic threat detection. His expertise lies in identifying and tracking malicious infrastructure through intricate domain analysis.

Beni Urech collaborated with Patrick Schläpfer in building the generic domain intelligence system, Shadow Spotter. His contributions largely focused on the architectural aspects of integrating AI agents with this platform. Beni's work emphasized the design of efficient agent structures, iterative tool development, and the utilization of protocols like MCP to enable seamless communication between LLMs and complex backend systems for advanced security analytics.