The domain takeover challenge: Detecting and defeating it at scale

The Domain Takeover Challenge: Detecting and Defeating It at Scale

Speakers: Ramesh (Security Engineer, Block), Eli F (Product Security Engineer, Block)

Conference: fwd:cloudsec

YouTube: https://www.youtube.com/watch?v=Ety04eFRFSU

Overview

In an increasingly complex cloud landscape marked by rapid growth, numerous acquisitions, and accelerated development cycles, organizations face significant challenges in managing their digital assets securely. This talk, delivered by Ramesh and Eli F from Block, addresses one such critical challenge: domain takeovers. The speakers delve into the intricacies of various domain takeover classes, their potential impact, and, most importantly, the scalable, event-driven system Block built to continuously detect and defeat these vulnerabilities across its vast and dynamic infrastructure.

The presentation underscores the transition from a reactive security posture, heavily reliant on bug bounty reports, to a proactive and automated system. For large organizations like Block, with multiple acquisitions and a culture of developer autonomy, the problem of orphaned DNS records and unmanaged domains can escalate rapidly. This article will explore the technical details of Block's solution, emphasizing the crucial role of ownership attribution and continuous synchronization in achieving a near-zero rate of domain takeover vulnerabilities.

Background

▶ Watch: Introduction to DNS takeovers at Block (0:45)

The genesis of Block's urgent focus on domain takeovers can be traced back to April 2021, when a single bug bounty researcher reported a startling 80 vulnerable domains. This incident served as a stark wake-up call, highlighting that simply enumerating public DNS records could expose a significant attack surface where Block-owned domains pointed to name servers no longer under their management. The sheer scale of the problem was exacerbated by Block's growth strategy, which included numerous acquisitions such as Afterpay, Tidle, and Weebly. Each acquisition brought with it not just new resources but also existing DNS problems, inheriting a fragmented and often unmanaged domain landscape.

Compounding this issue was Block's developer-centric culture, which encouraged rapid iteration in the cloud. Developers frequently configured CNAME records to point to third-party applications like GitHub and Heroku. While the underlying application resources might eventually be decommissioned or cleaned up, the associated DNS records often outlived them, creating "dangling" or "orphaned" entries. The fundamental problem with DNS, as highlighted by the speakers, is the difficulty in attributing ownership—who is responsible for a particular domain when there are multiple registrars, cloud accounts, and disparate teams involved?

The severity of this problem was evident in the bug bounty statistics. Between 2016 and 2019, Block received approximately 15 domain takeover bug bounty reports. This number surged dramatically to 79 reports between 2020 and 2022 (excluding the 80 found internally in 2021). By implementing the described solution, Block aimed to shift from a reactive stance, relying on external researchers, to a proactive ecosystem, ultimately achieving a near-zero rate of reports by the end of 2024 because issues were being detected much earlier in the pipeline.

Eli F then detailed the four primary classes of domain takeovers:

1. Name Server (or Hosted Zone) Takeovers: This is the most critical class, granting an attacker full control over a domain and its subdomains. It occurs when a domain's authoritative name servers are no longer managed by the legitimate owner, and an attacker can claim the zone. The analogy used was finding an empty, unlocked room (suite 400 on the fifth floor) after being directed there, allowing an attacker to put their name on the door and claim it. In DNS terms, a dig lookup for such a domain might return a "refused" status from the parent name server, indicating no knowledge of the domain, making it ripe for takeover.
2. CNAME Takeovers: This is a common scenario where a subdomain's CNAME record points to an external third-party service (e.g., S3 bucket, GitHub Pages) that has since expired or been decommissioned without the CNAME record being cleaned up. Attackers can then register the target service (e.g., an S3 bucket with the same name) and effectively take over the subdomain. Detection is complex because each third-party service returns a unique "fingerprint" when a resource is unavailable (e.g., "bucket does not exist" for S3, a 404 for GitHub Pages). Resources like Ed Overflow's GitHub repository provide valuable fingerprints for various services.
3. Elastic IP (or A Record) Takeovers: This occurs when an A record points to a specific IP address that is no longer controlled by the organization. For instance, an Amazon EC2 instance might be decommissioned, releasing its Elastic IP back into a shared pool. If a DNS A record still points to that IP, another user could claim it, effectively taking over the domain.
4. MX Takeovers: While less common, this type of takeover allows an attacker to intercept email traffic for the compromised domain by pointing its MX (Mail Exchanger) record to a mail server they control.

The impact of these takeovers ranges from complete control over a domain, enabling malicious content hosting and user redirection, to the ability to issue trusted TLS certificates for deceptive sites. Crucially, if a root domain has a session cookie scoped to all its subdomains, a subdomain takeover could allow an attacker to gain access to that session cookie, leading to broader compromise.

Detecting these issues involves using tools like dig. An NXDOMAIN response generally indicates the domain doesn't exist and is safe. A Server Failure is a moderate indicator of potential vulnerability, but could also be due to misconfiguration or network issues. A "refused" response, particularly for hosted zones, is a very high-confidence signal of a potential takeover. It's critical to query each name server for a domain, as DNS redundancy means records might not be identical across all servers.

Key Findings

▶ Watch: The 2021 bug bounty report incident (2:45)

Block's journey to combat domain takeovers yielded several significant findings and a robust, scalable solution:

• Shift from Reactive to Proactive: The organization successfully moved away from a reactive security model, where vulnerabilities were primarily discovered via bug bounty programs, to a proactive, automated detection system. This fundamental shift allowed them to identify and remediate issues much earlier.
• Scalable, Event-Driven Architecture: Block developed an event-driven system that continuously synchronizes DNS information from various sources (registrars, AWS, GCP, Cloudflare) with cloud asset data. This system is designed to scale from tens to tens of thousands of domains, proving its applicability across diverse organizational sizes.
• Criticality of Ownership Attribution: A core insight was that "detection without attribution eventually becomes shelfware." Understanding and clearly defining who owns a domain or DNS record is paramount for effective remediation. Block emphasized the creation of a RAIKI matrix and an ownership model.
• Registrar Consolidation as a Foundation: While extremely challenging and "painful," consolidating domains under a single registrar was identified as a crucial first step for establishing a clean baseline and enabling effective automation. This process often involves significant organizational and legal hurdles.
• Significant Improvement in Detection and Closure Rates: The implemented solution led to a 10x increase in detection rates, moving from approximately 2 detections per month to around 24 per month over two and a half years. This resulted in the detection of over 1,700 domain takeover issues and an impressive 92% vulnerability closure rate, indicating effective remediation or risk acceptance.
• Lessons Learned on Complexity: The team learned that attribution is rarely 100% perfect due to the distributed nature of DNS ownership. They also discovered that AWS account closures are a common and often overlooked cause of domain takeovers, as associated Route 53 records can become orphaned.
• Strategic Alerting and Validation: An effective alert strategy involved starting with all alerts as "hidden" until confidence in their accuracy was established. For transient issues like Elastic IP takeovers, delayed validation was implemented to prevent false positives and allow time for operational reuse.

Technical Deep Dive

▶ Watch: Introduction to the four domain takeover classes (4:50)

Block's approach to tackling domain takeovers evolved significantly, moving from rudimentary manual efforts to a sophisticated, automated system. Initially, an internal tool was developed to scan DNS records and generate reports for manual triage, a process that proved unsustainable at scale. Subsequent evaluations of external tools revealed issues with cost, integration, and limited scope. Even existing CSPM (Cloud Security Posture Management) platforms, while offering some DNS takeover detection, were typically scoped only to cloud resources, failing to cover the broader attack surface including external registrars. This led Block to develop a custom, comprehensive solution.

The fundamental challenge underpinning all these approaches was DNS ownership. In a large organization like Block, with multiple acquisitions and decentralized development, domains and their associated DNS records are often scattered across various registrars and cloud accounts. The example of Block acquiring, then selling, a company, only to find its name server records still pointing to Block's Route 53 accounts, perfectly illustrates this complexity. Without clear ownership, even detected issues become "shelfware"—identified but never resolved.

To address this, Block undertook several foundational steps:

1. Consolidation with a Single Registrar: Acknowledged as the "most painful thing," this was deemed essential to establish a clean, manageable baseline. This involved overcoming significant organizational and legal hurdles.
2. Domain Reconnaissance and Initial Cleanup: Before automation, a thorough inventory and cleanup of dangling DNS records were necessary, often involving difficult decisions about live resources with unknown ownership.
3. Vendor Tool Extension: Block leveraged an existing third-party vulnerability scanner acquired through an acquisition. This scanner was extended to integrate with their registrars and cloud resources, providing a unified view of their external attack surface.
4. Ownership Model and RAIKI Matrix: A clear RAIKI matrix (Responsible, Accountable, Consulted, Informed) and ownership model were established to ensure that every identified DNS takeover issue could be attributed to a specific team or individual for remediation.

The high-level architecture of Block's solution is designed around ingesting data from diverse sources into a central vulnerability scanner, then processing its outputs for alerting and attribution.

Data Sources include:

• Domain registrars (multiple initially, consolidated later).
• AWS (Route 53 records, EC2 Elastic IPs).
• GCP (Cloud DNS, App Engine).
• Cloudflare.

This information is fed into their vulnerability scanner. While Block uses a third-party scanner (due to an acquisition and its broader capabilities), the speakers noted that a simple set of scripts performing dig lookups could achieve similar results for hosted zone takeovers. The scanner's outputs include security alerts, an asset inventory of Block's owned domains, and initial attribution data.

The solution was built on four key design principles:

1. Event-driven: Alerts are generated and processed as soon as an issue is detected.
2. Continuously Synchronizing: Ensures the system always reflects the current state of DNS records.
3. Simple (Stateless): Minimizing state management reduces complexity and potential for errors.
4. Attribution: Ensuring alerts reach the right person or team for remediation.

Delving deeper into the data flow, asset sources (registrars, AWS, GCP) provide domain information to the vulnerability scanner, which continuously pulls an inventory. When the scanner detects a takeover issue, it sends a webhook event to a processing engine. This engine performs enrichment by calling internal APIs to gather more context about the asset (e.g., associated AWS account, team name). This enriched information is then used to attribute the alert to an owning party, which is then pushed to Block's vulnerability management platform for ticket creation and assignment to the responsible team.

The system architecture primarily leverages AWS components:

• An EventBridge service, on a schedule, triggers a Lambda function to pull DNS domain information from registrars. This Lambda scrubs the data, removes expired domains, and uploads valid domains to the vulnerability scanner.
• Native integrations exist for GCP and Cloudflare with the scanner, providing a single pane of glass for all DNS information.
• Another EventBridge instance, residing in an AWS organization account, listens for the creation of new AWS accounts. Upon detection, it triggers a cross-account trigger to the same Lambda, pushing new account information to the scanner, ensuring new assets are immediately onboarded.
• The vulnerability scanner integrates with cloud accounts using permissions (roles) to read network resources like EC2 instances, Route 53 zones, App Engine applications, and Cloud DNS configurations.

When the scanner identifies an exposure, it sends a message to an API Gateway. This gateway proxies the request to a Lambda Authorizer to verify the traffic originates from Block's tenant. Once authorized, a Translation Lambda performs the heavy lifting: it queries the vulnerability scanner for exposure type and remediation guidance, then integrates with internal systems to determine the resource owner. Finally, the alert is pushed to the vulnerability management platform, which handles the entire lifecycle of the vulnerability ticket based on this attribution.

A specific nuance in the architecture is the handling of Elastic IP takeovers. Because released Elastic IPs return to a shared pool and can be re-claimed for operational reasons, Block implemented a delayed validation mechanism. When the scanner detects an EIP takeover issue, it's routed through a Step Function. This Step Function introduces a waiting period of a few days. If the EIP is still exposed after this period, it's then fired off as a confirmed alert, reducing false positives.

Block's solution also ingeniously addresses the challenge of maintaining state without actually doing so within their custom components. The vulnerability scanner itself acts as the source of truth for the current state of exposures. If a user closes a ticket in the vulnerability management platform without actually resolving the underlying issue, the scanner, in its next continuous run, will re-detect the exposure and fire off a new webhook event, causing the vulnerability management system to simply recreate and reassign the ticket. Conversely, if an issue is genuinely resolved, the scanner will no longer detect it, triggering an auto-closure webhook. This stateless approach simplifies the system and guarantees continuous enforcement.

The attribution problem was a major focus. Block already had a mature ownership attribution system, which allowed them to route alerts intelligently. For instance, hosted zone takeovers, which require access to critical domain registrars, are statically routed to the network security team. In contrast, a GitHub Pages takeover might be routed directly to the application team responsible for the subdomain, as they have the context to decide whether to fix a misconfiguration or clean up the DNS record. Attribution is hard due to tooling limitations (scanners lack context) and operational complexity (distributed ownership models across legal, engineering, and application teams). Block's mechanisms for attribution include mapping AWS accounts to teams, using code attribution (commit history, owners files), and employing static defaults for critical infrastructure.

Demo / Proof of Concept

▶ Watch: Critical hosted zone takeovers explained with analogy (6:00)

While the talk thoroughly described the architecture, components, and operational flow of Block's internal domain takeover detection and remediation system, it did not feature a live, interactive demonstration of their specific platform or a proof of concept exploit. The speakers used dig command examples to illustrate how domain lookups reveal vulnerability indicators (like "refused" status) and referred to external resources like Ed Overflow's repository for CNAME takeover fingerprints. The focus was on the systemic solution rather than a specific tool's live functionality.

Defensive Implications

▶ Watch: Ed Overflow: A key resource for CNAME detection (8:50)

Organizations can draw several critical defensive implications from Block's comprehensive approach to domain takeovers:

• Implement Continuous, Event-Driven Monitoring: Establish a system that continuously synchronizes DNS records from all sources (registrars, cloud providers) with your asset inventory. This event-driven approach ensures vulnerabilities are detected as soon as they arise, moving beyond periodic scans.
• Prioritize Hosted Zone Takeovers: Recognize that hosted zone takeovers represent the highest impact vulnerability. Prioritize detection and remediation efforts for these issues, as they grant attackers full control over a domain.
• Establish Clear Ownership Attribution: Develop a robust RAIKI matrix and an ownership model for all domains and DNS records. Without clear attribution, even detected vulnerabilities will languish. Automate alert routing based on this ownership data to minimize triage bottlenecks.
• Consolidate Domain Registrars: While challenging, centralizing domain management under a single registrar significantly simplifies oversight, improves consistency, and streamlines security processes. Be prepared for the organizational and legal complexities involved.
• Proactive Cleanup of Dangling Records: Regularly audit and clean up orphaned DNS records, especially those pointing to decommissioned third-party services or cloud resources. This is particularly crucial after acquisitions, divestitures, or significant infrastructure changes.
• Leverage Delayed Validation for Transient Issues: For vulnerabilities like Elastic IP takeovers, implement delayed validation using mechanisms like a Step Function. This prevents alert fatigue from transient re-use of IPs and ensures only persistent exposures trigger alerts.
• Utilize Third-Party Safeguards: For CNAME records pointing to external services (e.g., GitHub Pages), leverage any domain verification or ownership safeguards offered by the third-party provider. This can prevent exploitation even if a CNAME record is inadvertently left dangling.
• Be Mindful of Cloud Account Closures: Understand that decommissioning cloud accounts (e.g., AWS accounts) can leave associated DNS records (like Route 53 entries) orphaned and vulnerable to takeover. Integrate DNS cleanup into account decommissioning processes.
• Strive for Automated Remediation and Proactive Checks: As a future state, aim to implement auto remediation for high-confidence issues (e.g., automated PRs to remove vulnerable records). Additionally, integrate proactive detection into developer workflows, prompting cleanup of DNS records before associated resources are deleted.
• Start Alerts as Hidden: When rolling out a new detection system, initially configure all alerts as "hidden" or in a testing phase. This allows for fine-tuning and building confidence in the accuracy of detections before flooding teams with potential false positives.

Key Takeaways

• Domain takeovers are a critical, scalable security problem for growing organizations, exacerbated by acquisitions and rapid developer iteration.
• Moving from reactive bug bounty reports to a proactive, continuous, and event-driven detection system is essential for effective mitigation.
• Clear ownership attribution is as vital as detection itself; without a well-defined RAIKI matrix and ownership model, identified issues will struggle to be resolved.
• Registrar consolidation and initial cleanup of existing DNS records, though challenging, form the foundational baseline for any scalable domain security program.
• Prioritize remediation efforts for hosted zone takeovers due to their highest potential impact, granting full control over a domain.
• Implement delayed validation for transient issues like Elastic IP takeovers to prevent alert fatigue and ensure only persistent vulnerabilities are flagged.
• Automating remediation and integrating proactive detection into development workflows represent the next frontier for enhancing domain takeover defense.

About the Speaker(s)

Ramesh is a Security Engineer on Block's platform security team. He brings a wealth of experience, having been with Block for over six years and possessing close to two decades of expertise across various domains including data center, network security, cloud, and Kubernetes.

Eli F serves as a Product Security Engineer within Block's product security engineering team. His work primarily focuses on the automated detection and remediation of code-based vulnerabilities at scale. Eli's personal interests lie in web application security, bug bounty programs, and root cause analysis. He is particularly drawn to solving complex problems within organizations that often fall between the cracks of individual teams, making domain takeovers a fitting area of focus for his expertise.