Imposter Detection with Watchman

Imposter Detection with Watchman

Speakers: Matthew Wollenweber, Security Operations, Incident Response, Exus

Conference: ShmooCon

YouTube: https://www.youtube.com/watch?v=DPOEzJCMWxs

Overview

In this ShmooCon presentation, Matthew Wollenweber introduces Watchman, an open-source tool designed to provide rapid and cost-effective detection of imposter domains. The talk addresses a critical gap in the cybersecurity landscape: the slow and prohibitively expensive nature of commercial brand monitoring services that often fail to alert organizations to malicious domain registrations until long after they've been weaponized. Wollenweber, drawing from his extensive experience in security operations and incident response, developed Watchman as a personal project to overcome these frustrations, particularly in the context of combating sophisticated phishing and Business Email Compromise (BEC) attacks.

Watchman's core innovation lies in its ability to leverage publicly available ICANN Zone files and process them with remarkable efficiency, identifying newly registered domains that mimic legitimate brands, including those of vendors and clients. This proactive approach aims to shift detection left in the attack kill chain, enabling defenders to identify potential threats at the moment of domain registration rather than weeks later when an attack is already underway. The tool's emphasis on speed, affordability, and comprehensive monitoring capabilities for critical attack indicators like MX records and active websites makes it a compelling alternative for organizations seeking to enhance their imposter domain detection strategies.

The importance of this work cannot be overstated. Phishing and BEC schemes, often facilitated by imposter domains, continue to be among the most financially damaging cyber threats, costing businesses millions of dollars annually. By providing a free, fast, and transparent solution, Watchman empowers a broader range of organizations, including those with limited budgets, to defend against these pervasive attacks more effectively. It challenges the status quo of commercial services, demonstrating that high-quality, timely imposter detection is achievable without exorbitant costs or significant delays.

Background

▶ Watch: Official start: Imposter Detection with Watchman agenda (8:00)

The problem of imposter domains is deeply rooted in the ease with which attackers can register lookalike domains to impersonate legitimate entities. These domains are primarily used for phishing campaigns, Business Email Compromise (BEC), and other forms of online fraud. Examples include domains that slightly misspell a company name (e.g., insomniac.tech instead of insomniac.com), use different top-level domains (TLDs), or add deceptive prefixes/suffixes (e.g., company-partner.com). While some such domains might be benign (e.g., branding, shadow IT, marketing campaigns), a significant portion are registered with malicious intent, often targeting accounts payable departments with fake invoices or attempting to harvest credentials.

Existing commercial services for imposter domain detection typically fall short in several critical areas, leading to significant frustration for security teams. Wollenweber highlights that these services often deliver alerts three to four days after an attack has been observed and up to three weeks after the malicious domain was initially registered. This substantial delay drastically reduces the window for proactive defense and increases the likelihood of successful attacks. Furthermore, these services are often expensive, with costs potentially reaching $10,000 per year per brand monitored, and crucially, they frequently focus solely on the client's own brand, neglecting the equally critical need to monitor for imposter domains targeting their vendors or clients, which are common vectors for BEC attacks.

The technical foundation for domain monitoring relies heavily on the Domain Name System (DNS). DNS translates human-readable domain names into IP addresses, facilitating internet communication. Key entities in this ecosystem include domain registrars (e.g., GoDaddy, Cloudflare) and Top-Level Domains (TLDs) like .com or .tech. Central to Watchman's operation is ICANN (Internet Corporation for Assigned Names and Numbers), the non-profit organization responsible for coordinating the global internet's DNS. ICANN provides access to Zone files, which are essentially lists of all registered domains within specific TLDs. While accessing these files requires a legitimate reason (such as brand monitoring or fishing detection) and can involve navigating administrative hurdles, especially for newer or "shady" TLDs, they offer a powerful, raw data source for monitoring. The .com TLD is by far the largest and most critical, containing hundreds of millions of domains, and thus is the primary focus for effective imposter detection.

Key Findings

▶ Watch: The impact: phishing and business email compromise (BEC) (9:30)

The central finding presented by Matthew Wollenweber is that the delays and high costs associated with commercial imposter domain detection services are largely unnecessary and can be overcome with an efficient, open-source approach. Watchman demonstrates that it is entirely feasible to detect newly registered imposter domains on the same day they appear in ICANN Zone files, drastically reducing the time-to-detection from weeks to hours or even minutes. This accelerated detection window provides a critical advantage for defenders, allowing for earlier intervention before malicious domains are fully weaponized and used in attacks.

Wollenweber's analysis reveals that the inefficiencies of commercial tools likely stem from their reliance on slow, "naive" processing methods, such as iteratively inserting millions of new domain records into a database. This brute-force approach, while functional, results in processing times that can stretch for a week or more for a single TLD like .com. Watchman's merge sort and diffing technique, by contrast, offers a significantly faster alternative. By comparing sorted lists of current and previous Zone files, Watchman can quickly identify only the newly registered domains, dramatically reducing the database load and processing time to mere minutes or hours, even on a consumer-grade laptop.

Another key finding is the strategic importance of monitoring specific indicators beyond mere registration. The presence of MX records (indicating email server configuration) and an active website (on ports 80 or 443) are strong signals that a newly registered imposter domain is being prepared for an active attack. Commercial services often fail to provide timely alerts on these changes, whereas Watchman is designed to continuously monitor and alert on these critical shifts in a domain's status. This proactive monitoring allows organizations to anticipate and prepare for phishing or BEC campaigns rather than reacting to them after they've launched.

Finally, the talk implicitly highlights that valuable security capabilities do not always require proprietary, expensive solutions. By making Watchman open-source and leveraging publicly available data (ICANN Zone files), Wollenweber demonstrates that robust imposter detection can be accessible and affordable. This empowers smaller organizations or those with limited budgets to implement sophisticated threat intelligence mechanisms that were previously out of reach, thereby leveling the playing field against well-resourced attackers.

Technical Deep Dive

▶ Watch: Why Watchman was built: Addressing existing service failures (11:50)

Watchman is an open-source, Python-based tool built on the Django web framework, designed for efficient and rapid detection of imposter domains. Its core functionality revolves around ingesting and processing ICANN Zone files, which are comprehensive lists of all registered domains under various TLDs. The most critical of these is the .com TLD, which alone contains over 200 million records.

The process begins with obtaining access to ICANN Zone files. Users can request free access to these root Zone files from the ICANN website, often by stating a purpose like brand monitoring. While some TLDs, particularly newer or "shady" ones, might impose additional hurdles or require more frequent renewals, the .com Zone file is generally accessible after an initial setup period of a few weeks. Watchman primarily extracts the domain name and its TLD from these files, disregarding the more detailed DNS record information for its initial processing.

Watchman's primary innovation in processing these massive datasets lies in its approach to identifying newly registered domains. A "naive" solution would involve downloading all Zone files, iterating through each unique domain, checking if it exists in a local database, and if not, inserting it. This method is incredibly slow, potentially taking a week to process the .com TLD alone due to the sheer volume of database inserts. Watchman employs a much more efficient technique:

1. Download Zone Files: It downloads the latest Zone file for each monitored TLD.
2. Sort and Diff: Instead of individual database inserts, Watchman sorts the newly downloaded file and compares it against the previously stored (and also sorted) file. This is essentially a merge sort or diffing operation.
3. Identify New Domains: By diffing the two sorted lists, Watchman quickly identifies only the domains present in the new file that were not in the old file. This list of new domains is significantly smaller than the total Zone file.
4. Batch Insert: These newly identified domains are then batch inserted into Watchman's PostgreSQL database, a process that takes minutes rather than days or weeks. This method dramatically reduces the computational overhead and makes the system viable for daily execution.

Once a new domain is identified, Watchman applies various search mechanisms to determine if it's an imposter:

• Substring Search: This is the simplest method, checking if a brand name (e.g., "Insomniac") appears as a substring within the new domain.
• String Edit Distance (Levenshtein Distance): This more sophisticated technique calculates the number of single-character edits (insertions, deletions, substitutions) required to change one string into another. Users can set a tolerance level (e.g., a distance of 1-3) to catch typosquatting or subtle variations. For example, insomnia.tech might have an edit distance of 3 from insomniac.com.
• Regular Expressions (Regex): For more complex pattern matching, Watchman supports regex searches, allowing for highly customized detection rules.

Beyond initial registration, Watchman continuously monitors matched domains for signs of active malicious intent:

• MX Record Monitoring: It checks for the presence of MX (Mail Exchange) records, which indicate that the domain is configured to send and receive email. The activation of MX records is a strong signal that an attacker plans to use the domain for phishing or BEC.
• Website Presence Monitoring: Watchman checks ports 80 (HTTP) and 443 (HTTPS) to determine if a website has been stood up on the domain or its www subdomain. The appearance of a website, especially one mimicking the legitimate brand, is another critical indicator of an impending attack.
• VirusTotal Enrichment: For identified imposter domains, Watchman integrates with VirusTotal to retrieve additional threat intelligence, such as threat severity and known malicious associations. This enrichment provides immediate context for analysts.

The system is designed for operational deployment, utilizing Celery workers to manage asynchronous tasks like Zone file downloads, processing, and continuous monitoring. While it can be self-hosted on a moderately powerful machine (e.g., a 2020 laptop with 16-32GB RAM), the speaker notes that hosting in cloud environments like AWS can be expensive due to the significant data transfer volumes (hundreds of gigabytes daily) involved in downloading Zone files.

Alerting capabilities are robust, with support for Slack webhooks (sending JSON payloads) and email notifications. Users configure search parameters and alert destinations through a Django admin panel. Watchman also allows for managing identified matches, enabling users to mark false positives or ignore specific domains to refine future alerts. Future development includes interactive Slack messages, automated rule generation from email inboxes, and ephemeral cloud-based screenshotting of potentially malicious websites.

Demo / Proof of Concept

▶ Watch: Technical dive: How domain monitoring utilizes DNS (12:50)

While the talk did not feature a live, interactive demonstration, Matthew Wollenweber provided a clear walkthrough of Watchman's functionality using screenshots from its Django admin panel and an example of a Slack alert. This served as a compelling proof of concept for the tool's capabilities.

The core of the "demo" showed how users configure Watchman:

1. Configuring a Search: A screenshot illustrated the Django admin interface where a user can define a search. This involves inputting the target string (e.g., a brand name), selecting the type of search (e.g., substring, string edit distance, or regex), and for string edit distance, specifying a tolerance level. This panel demonstrates the flexibility in defining what constitutes an "imposter" based on different matching criteria.
2. Configuring an Alert: Another screenshot displayed the alert configuration panel. Here, users can set up destinations for alerts, primarily Slack webhooks or email. The speaker noted that he mainly uses Slack webhooks, which involve sending a JSON payload to a configured URL, showcasing the simplicity of integration with common communication platforms.

The most impactful part of the demonstration was an actual Slack alert generated by Watchman during the conference itself. This alert, triggered by a newly registered domain, provided several key pieces of information:

• The domain name that matched a configured search.
• The creation date of the domain (e.g., "created on the 7th").
• Enrichment details, such as the registrar (e.g., Namecheap, often associated with malicious registrations).
• The status of MX records (e.g., has MX records: True), indicating potential email activity.
• The status of a website (e.g., has website: False).
• A link to view the match details via Watchman's internal API, which provides the raw JSON data including VirusTotal enrichment (e.g., threat severity, volume).

This real-world example effectively showcased Watchman's ability to provide timely, detailed, and actionable intelligence. The alert explicitly stated whether the domain had MX records, a crucial indicator for imminent BEC or phishing attacks. Furthermore, the ability to edit match statuses within the Django admin panel (e.g., marking a false positive or ignoring a benign match) was highlighted, demonstrating the tool's practical usability for refining detection. The "demo" thus successfully conveyed Watchman's ease of configuration, its rapid detection capabilities, and the richness of the information it provides to defenders.

Defensive Implications

▶ Watch: Leveraging ICANN and root zone files for data (13:30)

Watchman offers several significant defensive implications, fundamentally shifting the paradigm for imposter domain detection from reactive to proactive.

1. Early Warning System: The most critical implication is the ability to detect imposter domains on the day of registration. This dramatically shortens the window of vulnerability, allowing defenders to identify potential threats weeks before commercial services would typically alert them. This early warning enables organizations to initiate takedown procedures, block domains, and educate users before attacks are fully launched.

1. Comprehensive Monitoring for BEC/Phishing: By actively monitoring for the presence of MX records and active websites on newly registered imposter domains, Watchman provides direct indicators of an attacker's intent to launch email-based phishing or BEC campaigns. Defenders can prioritize these domains for immediate investigation and mitigation, focusing resources on the highest-risk threats.

1. Protection for Vendor and Client Ecosystems: Unlike many commercial services that focus solely on an organization's primary brand, Watchman allows for monitoring any string. This capability is crucial for protecting against BEC attacks that often impersonate trusted third parties (vendors, clients, partners). By monitoring these external brands, organizations can proactively identify threats that could impact their supply chain or business relationships.

1. Cost-Effective Threat Intelligence: As a free and open-source tool, Watchman democratizes access to advanced imposter detection capabilities. Organizations with budget constraints can now implement robust brand monitoring without incurring the prohibitive costs associated with commercial services. This allows for a broader and deeper scope of monitoring, covering more brands and TLDs than might otherwise be affordable.

1. Actionable Intelligence and Integration: Watchman's alerts, delivered via Slack or email, provide detailed information including registrar, MX record status, website status, and VirusTotal enrichment. This rich context is immediately actionable for incident response teams. The ability to output public blocklists via JSON also facilitates integration with existing security tools like SIEMs (Security Information and Event Management) or SOAR (Security Orchestration, Automation, and Response) platforms, enabling automated blocking of malicious domains.

1. Empowering Internal Security Teams: By providing direct access to ICANN Zone file data and a transparent processing methodology, Watchman empowers internal security teams to understand and control their imposter detection strategy. This reduces reliance on opaque third-party vendors and fosters a more hands-on, informed approach to threat intelligence.

While Watchman primarily focuses on ICANN-managed TLDs, defenders should also consider integrating its output with other data sources, such as commercial passive DNS feeds (if budget allows for "insanely expensive" but "great" ones like Farsight) to catch imposter domains in non-ICANN TLDs or those that might not immediately appear in Zone files but are observed in DNS queries. However, for the vast majority of imposter domains, Watchman provides a powerful and accessible first line of defense.

Key Takeaways

• Commercial imposter domain detection services are often slow and expensive, providing alerts weeks after domain registration and days after attacks begin, often costing upwards of $10,000 per brand per year.
• Watchman offers a free, fast, and open-source alternative that can detect newly registered imposter domains on the same day they appear in ICANN Zone files, significantly reducing the detection window.
• The tool achieves its speed by leveraging ICANN Zone files and an efficient merge sort/diffing technique to identify new domains, avoiding slow, iterative database inserts.
• Watchman monitors for critical indicators of attack readiness, including the presence of MX records (email setup) and active websites (on ports 80/443), providing timely alerts when these statuses change.
• It supports various search methodologies like substring matching, string edit distance (for typosquatting), and regex, enhanced with VirusTotal enrichment for comprehensive threat intelligence.
• Watchman empowers organizations to proactively monitor imposter domains targeting their vendors and clients, addressing a common and costly vector for Business Email Compromise (BEC) attacks that commercial services often overlook.

About the Speaker(s)

Matthew Wollenweber is a cybersecurity professional with a focus on security operations and incident response. He currently works at Exus, applying his expertise to defend against various cyber threats. Watchman is a personal project developed by Matthew, born out of his frustration with the limitations and inefficiencies of existing commercial solutions for imposter domain detection that he encountered in his professional capacity. His experience in handling incidents, particularly those involving Business Email Compromise (BEC) and phishing, motivated him to create a more effective, transparent, and accessible tool for the security community.