Patience brings prey: lessons learned from a year of threat hunting in the cloud

Patience Brings Prey: Lessons Learned from a Year of Threat Hunting in the Cloud

Speakers: Greg Foss, Anthony Randazzo

Conference: fwd:cloudsec North America 2025

YouTube: https://www.youtube.com/watch?v=cugwipMXHp0

Overview

Greg Foss and Anthony Randazzo from Datadog's product detection engineering team presented a year-in-review of their cloud threat hunting program, sharing operational methodology, two detailed case studies, and aggregate findings from investigating threats across thousands of customer environments. The talk bridged the gap between traditional hypothesis-driven threat hunting and the signal-driven, large-scale approach that a security vendor with massive telemetry can execute. Key findings included a MIMO/Mimolet threat actor employing rootkits and nine-day dwell times on compromised workloads, widespread LLM/Bedrock abuse via stolen access keys, and the unsurprising-but-quantified confirmation that compromised long-term access keys remain the dominant cloud breach vector.

Background

▶ Watch: Why Datadog threat hunts: protecting customers and discovering new techniques (2:00)

Traditional threat hunting is typically hypothesis-driven: a hunter asks a question, gathers relevant data, and investigates. Datadog's threat hunting program operates differently due to its scale -- analyzing security signals across thousands of environments from a vendor vantage point. Their primary inputs include critical signal analysis (sampling high-fidelity security alerts across customer environments), detection maintenance cycles (improving existing detection rules, which sometimes surface new activity), and targeted hypothesis-driven hunts from a maintained backlog. The program's outputs feed directly into Datadog's security product detection engineering, creating a loop where hunting findings become product detections.

Key Findings

▶ Watch: MIMO threat actor: 9-day dwell time, rootkit, and proxy jacking on Magento (9:30)

Compromised long-term access keys are the #1 cloud breach vector, by far. Behind that, business email compromise (BEC) is the second most common intrusion vector, characterized by attackers creating mail rules to hide emails from security teams (using rules named with a single period or spaces). Third is web application compromise through credential stuffing against both applications and identity providers like Okta where MFA is not configured. Endpoint compromise (info stealers, credential file scraping) rounds out the top vectors, though Datadog often learns about these through post-incident communication with affected customers rather than direct detection.

Case Study 1: MIMO/Mimolet Threat Actor. Discovered during detection tuning for a noisy rule about image file downloads, this case involved a workload compromised through a PHP FastCGI exploit targeting a Magento installation (likely via a third-party plugin). What made this case unusual was the nine-day dwell time between initial compromise and crypto miner deployment -- most workload compromises are smash-and-grab operations. The attacker deployed a rootkit called Diamorphine (referred to as "lambdar" in the talk) that uses Linux syscalls to hide executables in memory and from filesystem listings. They also established proxy jacking to monetize traffic, set up three separate C2 servers for persistence, and demonstrated tradecraft beyond what had been previously documented for this threat group (Sequoia had published prior research on MIMO targeting CraftCMS).

Case Study 2: LLM Abuse via Stolen Access Keys. Building on observed Bedrock abuse dating to early 2024 and community research (including Permiso's high-interaction honeypot), Datadog ran a hypothesis-driven hunt across their CloudTrail data looking for abuse of AI services across AWS, Azure, and GCP. They found that Azure and GCP AI service abuse was minimal compared to AWS Bedrock, theorizing this correlates with market share. The hunt cast a wide net but returned only two hits across thousands of environments -- both confirmed malicious. The compromised access keys were used for Bedrock API reconnaissance (list-models), ECS cluster creation for crypto mining, and IAM user creation for persistence.

Technical Deep Dive

▶ Watch: Pivoting tradecraft: expanding from one indicator across thousands of environ... (15:00)

The Datadog team's threat hunting methodology distinguishes between threat detection (identifying known activities with high-efficacy static rules) and threat hunting (finding unknown behaviors through wide-net behavioral analysis). For detection, they look for specific indicators like user agent strings, event names, and known attack patterns. For hunting, they analyze general user agent distributions, unusual access patterns, and anomalous region usage -- an underused signal, since many cloud environments have regions that have never been used, making any activity there highly suspicious.

Their pivoting tradecraft for cloud investigations starts with a single indicator (e.g., a compromised access key) and expands through associated VPN provider IPs, unique IAM usernames created for persistence, and even malicious AWS accounts (though these tend to be short-lived as AWS terminates them). Each indicator is then recast across the entire customer base to find additional compromises, creating a chain of discoveries.

The team runs quarterly hunting parties bringing together people from diverse security disciplines across Datadog -- not just the threat hunting team but internal security and other groups. These focused events assign specific hunt areas (like "find abuses of our platform" or "look into LLM abuse at scale") and pair people who don't normally work together.

A notable operational finding: attacker infrastructure is remarkably stable. Contrary to what the Pyramid of Pain might suggest about atomic indicators being easily changed, many threat groups reuse the same infrastructure and TTPs over extended periods, making indicator-based hunting more effective than expected in the cloud context.

Demo / Proof of Concept

▶ Watch: Top breach vectors: access keys #1, BEC #2, web app compromise #3 (16:30)

No live demo was performed. The talk presented findings and methodology from Datadog's operational threat hunting program, with charts showing detection gap improvements over four quarters.

Defensive Implications

▶ Watch: AI-augmented threat hunting and the detection feedback loop (21:00)

The overwhelming dominance of compromised long-term access keys as the breach vector reinforces the urgent need to eliminate them wherever possible in favor of short-lived credentials and role assumption. Organizations should monitor for unusual region activity as a high-signal, low-noise hunting indicator. BEC-related mail rule creation (especially rules with minimal or blank names) should be a top-priority detection. For workload security, the nine-day dwell time in the MIMO case demonstrates that crypto mining is not always immediate -- defenders need runtime monitoring that persists beyond initial compromise detection. The stability of attacker infrastructure means that even "low-value" IP-based indicators remain useful for cloud threat hunting longer than conventional wisdom suggests.

Key Takeaways

• Compromised long-term access keys remain the #1 cloud breach vector; BEC is #2; web application compromise is #3
• The MIMO threat actor showed nine-day dwell times, rootkit deployment (Diamorphine), and proxy jacking alongside crypto mining on compromised Magento workloads
• LLM/Bedrock abuse is concentrated on AWS; Azure and GCP AI services see minimal targeting
• Unusual region activity is a high-signal hunting indicator for cloud environments
• Attacker infrastructure is more stable than expected -- atomic indicators retain hunting value in cloud contexts
• Detection maintenance (tuning existing rules) is a productive source of new threat discoveries

About the Speaker(s)

Greg Foss manages the product detection engineering team at Datadog, focusing on workload security, SIEM, and threat hunting. He has approximately 15 years of experience across multiple organizations. Anthony Randazzo leads Datadog's detection engineering organization for security products, with about 20 years in cybersecurity including SOC analysis at DoD, incident response, cyber threat intelligence, and managed detection and response. He has been at Datadog for three years.