Boosting URL detection with syntactic features in spam emails

Boosting URL detection with syntactic features in spam emails

Speakers: Antonia Scherz, Data Scientist, Net at Work

Conference: Virus Bulletin

YouTube: https://www.youtube.com/watch?v=K5PLKwLGBX8

Overview

In the ever-evolving landscape of email security, detecting malicious URLs remains a critical challenge. Antonia Scherz, a Data Scientist at Net at Work, presented a focused talk at Virus Bulletin, delving into a novel approach for identifying nefarious links within email campaigns. Her presentation, "Boosting URL detection with syntactic features in spam emails," zeroes in on the often-overlooked syntactic structure of URLs to enhance detection speed and coverage, particularly for newly emerging threats. The core problem addressed is the inherent delay in traditional domain-level detection methods, which allows malicious emails to bypass defenses during a crucial initial window.

Scherz, part of Net at Work's 32 guards data team, highlighted the pervasive nature of email-borne threats, exemplified by the rampant "crypto chat trader" scams leveraging celebrity endorsements. These campaigns, designed to defraud victims of significant sums, necessitate immediate and robust detection mechanisms. The talk introduces a method that bypasses the need for extensive contextual data by analyzing the structural patterns of URLs themselves, making it particularly effective in environments where access to email content is restricted due to privacy concerns. This approach not only promises faster identification of new malicious domains but also offers a more explainable and less computationally intensive alternative to complex machine learning models.

The significance of this work lies in its ability to proactively identify entire spam campaigns, rather than reacting to individual malicious domains. By extracting regex patterns from URL structures, Scherz demonstrated how security teams can link seemingly disparate domains to a common threat actor, thereby painting a complete picture of an ongoing attack. This campaign-level intelligence enables defenders to block a broader spectrum of malicious URLs from the outset, significantly reducing the window of vulnerability and preventing potential financial losses for users.

Background

▶ Watch: Introduction and the malicious link problem (0:00)

The pervasive threat of malicious links embedded in spam and phishing emails demands sophisticated detection mechanisms. Traditional approaches often fall into two main categories: highly complex, character-level neural networks (like transformer models) or simpler, domain-based detection systems. While neural networks can be powerful, they typically suffer from a lack of explainability, making it difficult for security providers to justify blocking decisions to customers. Domain-based detection, on the other hand, is more transparent but faces a critical limitation: it requires a certain amount of aggregated data about a domain before it can confidently classify it as malicious.

This need for aggregated data – such as how many emails a domain has been seen in, how long it has been active, or its associations with trusted senders – introduces a delay. Even a few minutes of latency can be detrimental, allowing initial waves of malicious emails to reach inboxes before a new domain is flagged. Threat actors exploit this "analysis window" by rapidly cycling through new domains, ensuring that by the time one domain is blacklisted, another is already operational. This constant regeneration of URLs means that traditional blacklisting and domain reputation systems are always playing catch-up. The challenge is further compounded by strict data protection regulations (like GDPR) that prevent email security providers from accessing or storing the actual content of emails. This necessitates detection methods that rely solely on email metadata rather than message body analysis.

The specific context for Net at Work, as outlined by Antonia Scherz, involves processing millions of URLs daily using only metadata. This metadata includes information such as the mail source IP, Mail From and Header From fields, email authentication checks like SPF and DKIM, CSA (Certified Senders Alliance) whitelist status, and file hashes from attachments. Within this constraint, the goal is to achieve immediate detection without relying on content or incurring delays for aggregated domain reputation data. The talk uses the example of a prevalent "crypto chat trader" scam in Germany, which employs well-known celebrities to lure victims into fraudulent investment schemes, highlighting the real-world impact of these undetected malicious links and the urgent need for faster, more comprehensive detection.

Key Findings

▶ Watch: Introducing syntactic regex patterns for URL analysis (6:30)

The central discovery presented in this talk is the profound impact of leveraging syntactic features of URLs, specifically through the extraction of regex patterns, to significantly enhance the speed and coverage of malicious URL detection in email security. This method allows for a shift from reactive, domain-level blocking to proactive, campaign-level detection, addressing the critical gap where new domains associated with ongoing spam campaigns often evade initial defenses.

A primary finding is that by extracting a generalized regex pattern from the path and query components of URLs, security systems can rapidly identify and link newly encountered domains to known malicious campaigns. This capability dramatically reduces the "analysis window" during which fresh domains associated with a campaign might bypass detection. The speaker demonstrated how this approach effectively groups domains that share a common underlying structure, even if they originate from different mail source IPs—a common evasion tactic used by threat actors to distribute their load and avoid detection.

The most compelling quantitative result shared was that this regex pattern-based technique helped catch an impressive 90% of the URLs missed by traditional domain detection methods. This substantial improvement underscores the efficacy of analyzing URL syntax as an early-stage detection mechanism, particularly for fast-flux domains and rapidly evolving campaigns that generate new URLs frequently. By identifying the characteristic patterns of a campaign from its initial instances, subsequent malicious URLs conforming to that pattern can be blocked immediately, right from their first appearance. This not only bolsters security but also enables more efficient resource allocation by consolidating multiple specific detections into a single, broader campaign signature.

Technical Deep Dive

▶ Watch: Impact: Faster detection of new campaign domains (10:00)

The technical core of Antonia Scherz's presentation revolves around overcoming the limitations of traditional domain detection and content-based analysis by focusing on the syntactic structure of URLs. The foundational constraint for Net at Work's 32 guards team is the lack of access to email content due to data protection regulations. Therefore, all detection must rely exclusively on email metadata. This metadata includes critical elements such as the mail source IP, Mail From and Header From fields, SPF and DKIM authentication results, CSA whitelist status, all URLs found within the email body and attachments, and file hashes of any attachments. The system processes millions of URLs continuously, demanding immediate and explainable detection.

Traditional domain detection, while explainable, suffers from a latency problem. It requires accumulating aggregated data—such as domain frequency, age (time since first seen), and relationships with trusted senders—before a confident classification can be made. This process takes "a few minutes," creating a critical window during which newly launched malicious domains can bypass filters. More advanced methods involving character embeddings and transformer models offer high accuracy but lack the crucial explainability required for operational security, where every block needs a clear justification.

To bridge this gap, the talk introduces the concept of extracting regex patterns from the lateral part of URLs. This involves a systematic decomposition of the URL into its meaningful components and then generalizing the variable parts. Specifically:

1. Preservation of Meaningful Characters: Key structural elements of a URL, such as slashes (/), question marks (?), and equal signs (=), are retained as literals in the regex pattern.
2. Character Type and Length Description: The variable string segments between these meaningful characters are described by their character types (e.g., lowercase letters, uppercase letters, numbers, special characters) and their respective lengths. For instance, a segment like example123 might be represented as [a-z]{7}[0-9]{3} in a more granular regex, or more simply as (LOWERCASE|NUMBER){10} in the speaker's conceptual representation.
3. Special Handling for Fragments: The fragment part of a URL (after a #) is specifically analyzed for email patterns, as these are frequently used to pre-fill login forms in phishing attempts. This allows for targeted detection of this common tactic.

The output of this process for a given URL is a generalized regex pattern. For example, a URL like https://example.com/path/to/resource?param=value#[email protected] would be transformed into a pattern that describes the sequence of character types and lengths for each segment, while retaining the /, ?, =, and # characters. This pattern then serves as a signature for an entire spam campaign, rather than just a single domain.

A key advantage of this campaign-level detection is its ability to group related domains even when threat actors employ load distribution across many mail source IPs to evade reputation-based blocking. By identifying a common syntactic pattern, the system can link these disparate domains back to the same malicious operation. The speaker illustrated this with a timeline graph, showing how initial detections of a campaign's URLs (red dots) generate a regex pattern that then immediately catches all subsequent, newly generated domains (blue dots) belonging to the same campaign, thereby drastically reducing the attack window.

Further refinement of this technique addresses the problem of multiple regex patterns describing the same spam campaign. To reduce computational overhead and enhance generalization, the system attempts to group these similar regex patterns. An initial, simpler approach of merely erasing length descriptions from the patterns proved problematic, leading to false positives by matching benign, generic URLs.

The refined strategy involves:

1. Pre-grouping: URLs that are already likely very similar (e.g., sharing initial regex patterns) are grouped together. This step makes the subsequent analysis computationally feasible, as direct string comparison of millions of URLs would be intractable.
2. En-gram Extraction: From these pre-grouped URLs, en-grams (sequences of shared characters) are extracted. These common string segments act as more specific identifiers than generalized regex patterns, helping to filter out false positives.
3. False Positive Mitigation: The talk emphasized the critical importance of minimizing false positives, as customer complaints about legitimate emails being blocked are a major concern. A specific heuristic was identified: false positive rates were "acceptably low" when regex patterns resolved to registers with more than 50 tokens. A token is defined as either a special character with meaning from a URL, a part describing the character type in a string, or a length description. While acknowledged as "quite restrictive," this threshold proved effective because many observed spam campaigns utilize "very long and descriptive URLs."

The overall pipeline involves: initial domain detection -> regex pattern extraction (highly effective) -> regex grouping and en-gram filtering (effective for specific cases). Looking ahead, the team is exploring the application of machine learning to detect these regex patterns, recognizing the ongoing challenge of acquiring clean, labeled data for training.

Demo / Proof of Concept

▶ Watch: Managing false positives with regex token count (17:30)

While the talk did not feature a live, interactive demonstration, Antonia Scherz effectively provided a proof of concept through a detailed case study of an active "crypto chat trader" spam campaign. This demonstration illustrated the core problem the research addresses and the efficacy of the proposed regex pattern detection.

The speaker began by showcasing a deceptive link from the campaign, which, upon initial inspection or a VirusTotal lookup, might appear suspicious but not immediately reveal its full malicious intent. The link initially redirected to a seemingly random news page, a common tactic to bypass basic URL reputation checks. However, dissolving the full redirect path eventually led to the actual scam: a "GPT trader" platform promising quick riches, often endorsed by well-known celebrities in Germany to enhance perceived legitimacy. This real-world example underscored the financial danger these links pose.

The central demonstration of the regex pattern's impact was presented through a visual timeline. This graph displayed various domains associated with the crypto scam, with red dots marking the first time a domain was seen and blue dots indicating its subsequent reoccurrence. The timeline vividly illustrated a key characteristic of sophisticated spam campaigns: domains are often short-lived and rapidly regenerated to evade blacklists.

The critical insight from this demonstration was how the regex pattern detection addresses this rapid domain churn. By identifying the characteristic syntactic pattern from the initial few malicious URLs (represented by the red dots), the system could then generate a robust regex signature. This signature was then immediately applied to all subsequent incoming URLs. As a result, every new domain launched by the campaign (the blue dots) that matched the established regex pattern was detected and blocked right from the start, significantly reducing the campaign's effective lifespan and the number of malicious emails reaching users. This visual evidence powerfully conveyed how the method reduces the "analysis window" from minutes to near-instantaneous detection for new, related domains, effectively catching the 90% of URLs that traditional domain-based detection might initially miss. The presentation thus served as a compelling conceptual and results-driven demonstration of the technique's practical value.

Defensive Implications

▶ Watch: Key takeaways and future machine learning applications (19:50)

The insights presented by Antonia Scherz offer several crucial implications for organizations and security professionals involved in email defense. The primary takeaway is the urgent need to integrate syntactic URL analysis into early-stage detection pipelines, especially in environments where content inspection is restricted or delayed.

1. Accelerated Detection for New Threats: The most significant defensive benefit is the ability to detect newly emerging malicious domains almost instantaneously. By reducing the reliance on aggregated domain reputation data, which inherently introduces a latency of "a few minutes," organizations can block initial waves of highly targeted spam and phishing attacks that exploit this window. This proactive stance is critical for combating fast-flux campaigns and those that rapidly cycle through domains.

1. Campaign-Level Intelligence: Defenders should shift their focus from merely blocking individual malicious domains to identifying and neutralizing entire spam campaigns. The regex pattern approach provides the means to link disparate domains that share a common structural signature, even if they originate from different IPs. This allows security teams to gain a holistic view of an attacker's infrastructure and tactics, enabling more comprehensive and resilient defenses.

1. Explainable and Efficient Blocking: The use of regex patterns offers a clear, explainable basis for blocking decisions, which is vital for enterprise email security providers who need to justify actions to customers. Furthermore, by consolidating multiple specific URL variants into a single regex signature, the method reduces the computational overhead associated with maintaining extensive blacklists and performing individual domain lookups.

1. Robustness Against Evasion: Threat actors frequently distribute their malicious URLs across numerous mail source IPs to evade reputation-based filtering. The syntactic analysis technique can still group these distributed URLs, making it more resilient against such evasion tactics than IP- or domain-centric methods alone.

1. Strategic Application of En-grams: While powerful, the generalization through en-grams requires careful implementation to avoid false positives. Defenders should be mindful of the trade-off between broader detection and the risk of blocking legitimate emails. The heuristic of requiring regex patterns to resolve to registers with more than 50 tokens provides a valuable guideline for achieving an acceptably low false positive rate, suggesting that longer, more complex URL patterns are safer candidates for generalization. Organizations should tailor this threshold based on their specific risk appetite and the characteristics of their email traffic, potentially focusing initial deployments on new or highly suspicious traffic.

1. Augmentation, Not Replacement: This method is best seen as an augmentation to existing security layers, not a replacement. It excels at early, rapid detection of campaign structures, complementing traditional domain reputation, content analysis (where permissible), and sandbox detonation techniques. Incorporating syntactic features upstream in the detection process can filter out a significant volume of threats before they reach more resource-intensive analysis stages.

Key Takeaways

• Syntactic URL Analysis: Focusing on the structural patterns of URLs via regex patterns enables rapid detection of malicious links without relying on email content or delayed reputation data.
• Campaign-Level Detection: This approach effectively links related domains across an entire spam campaign, even when attackers distribute traffic across multiple mail source IPs, providing a comprehensive view of threats.
• Reduced Analysis Window: By identifying characteristic URL patterns from initial malicious instances, subsequent new domains from the same campaign can be detected and blocked immediately, significantly reducing the window of vulnerability.
• Significant Detection Boost: The technique proved highly effective, catching 90% of URLs that traditional domain-based detection methods initially missed, demonstrating its value as a front-line defense.
• En-gram Refinement: Grouping similar regex patterns and extracting en-grams helps generalize detection and reduce computational load, though careful application is needed to minimize false positives, especially for shorter, generic URLs.
• False Positive Control: A critical finding for practical deployment is that regex patterns resolving to registers with more than 50 tokens yield acceptably low false positive rates, guiding the application of the generalization technique.

About the Speaker(s)

Antonia Scherz is a Data Scientist at Net at Work, a German company specializing in email security solutions provided via their no spam proxy. She is an integral part of the 32 guards team, which is dedicated to analyzing email metadata, developing detection mechanisms, and managing the vast amounts of data involved. Scherz is passionate about email security and is actively involved in catching the latest threats, bringing a practical, data-driven approach to combating malicious campaigns.