Real world detection engineering in a multi-cloud environment
Aaron Jewitt (Detection Engineering Team Lead · Elastic)
TROOPERS23 · Day 1 · Defense & Management · Defense & Management
Overview
In his TROOPERS talk, "Real world detection engineering in a multi-cloud environment," Aaron Jewitt, Detection Engineering Team Lead at Elastic, delivers a compelling and practical look into the challenges and solutions for securing a globally distributed, cloud-native workforce. Jewitt, leveraging over two decades of experience spanning NSA's red team (TAO) and extensive work in threat hunting, articulates the stark realities that traditional security models struggle to address. The talk is not merely theoretical; it's a candid exploration of the "messy" real world of security operations, where perfect solutions are elusive, and continuous adaptation is paramount.

Key moments
- 2:40 What is "Real World" Detection Engineering?
- 4:20 Core principles: actionable, focused, context, zero false positives
- 10:30 Identity is the perimeter; why 502 MFA is crucial
- 14:00 All data in SIEM, ECS, and automated alert enrichment
- 20:30 The power of a robust asset inventory for detection
- 26:00 Cloud challenges: immense scale, speed, and inherent complexity
- 40:00 Detecting compromised tokens with automated workflow
- 48:30 Q&A: Log storage strategy and ML for human detection
Real world detection engineering in a multi-cloud environment
Speakers: Aaron Jewitt, Detection Engineering Team Lead, Elastic
Conference: TROOPERS
YouTube: https://www.youtube.com/watch?v=IHDNkpAr5JY
Overview
In his TROOPERS talk, "Real world detection engineering in a multi-cloud environment," Aaron Jewitt, Detection Engineering Team Lead at Elastic, delivers a compelling and practical look into the challenges and solutions for securing a globally distributed, cloud-native workforce. Jewitt, leveraging over two decades of experience spanning NSA's red team (TAO) and extensive work in threat hunting, articulates the stark realities that traditional security models struggle to address. The talk is not merely theoretical; it's a candid exploration of the "messy" real world of security operations, where perfect solutions are elusive, and continuous adaptation is paramount.
Jewitt's presentation is particularly relevant in an era where organizations increasingly operate with remote teams and rely heavily on multi-cloud infrastructure. He meticulously dissects the shift from network-centric perimeters to identity as the new security boundary, and the unique complexities introduced by the sheer scale, speed, and API-driven nature of cloud environments. His insights provide a crucial roadmap for security practitioners grappling with overwhelming data volumes, rapid attacker movements, and the constant evolution of cloud technologies, emphasizing the need for actionable, context-rich, and highly automated detection strategies.
The core message resonates deeply with anyone tasked with protecting modern enterprises: default rules are insufficient, resources are limited, and security often contends with business priorities. Jewitt champions a pragmatic approach, advocating for tightly focused detections, comprehensive data centralization, and aggressive automation to empower security teams to respond effectively against sophisticated threats. This talk serves as an essential guide for building resilient detection capabilities that can withstand the rigors of a dynamic and distributed threat landscape.
Background
▶ Watch: What is "Real World" Detection Engineering? (2:40)
Detection engineering, a relatively nascent but critical role in cybersecurity, involves crafting queries and rules to identify malicious activities or threat actor behaviors, triggering alerts when such conditions are met. Jewitt categorizes detection engineers into three main types: product-focused, threat intelligence-driven, and "real-world." Product-focused engineers prioritize low false positives for commercial security products. Threat intelligence engineers often create overly broad rules based on Indicators of Compromise (IOCs) from specific threat actors. Aaron Jewitt, however, identifies with the "real-world" detection engineer—a role often born from SOC analysts tired of excessive noise or red team engagements that went undetected. This type of engineer builds custom rules tailored to their specific environment, acknowledging that default security rules rarely suffice in complex, operational settings.
The "real world" is inherently messy. Security is frequently a secondary consideration to business needs, operating with limited resources. Pervasive challenges include the complexities of API keys, tokens, and identity management, the prevalence of Shadow IT (unauthorized systems), the maintenance of outdated legacy architecture, and the constant stream of noisy software updates from vendors like Lenovo, Dell, and HP that often mimic or obscure malicious activity. Furthermore, integrating diverse third-party providers with disparate log formats adds layers of complexity.
For Elastic, a company that has been globally distributed for over a decade with employees in 42 countries, the traditional network perimeter is nonexistent. There's no corporate LAN, no file shares, and no special privileges behind a firewall. This distributed model, while offering advantages like the absence of Active Directory or Exchange servers, necessitates a complete re-evaluation of security strategy. The perimeter shifts from network boundaries to identity. In this environment, security teams must contend with a vast array of cloud-native applications and third-party systems, collecting enormous volumes of data—much of it from their own cloud SaaS offerings. This backdrop sets the stage for Jewitt’s principles, which are designed to bring order and efficacy to this chaotic reality.
Key Findings
▶ Watch: Identity is the perimeter; why 502 MFA is crucial (10:30)
Aaron Jewitt outlined several core principles for effective real-world detection engineering in a distributed, multi-cloud environment, emphasizing pragmatism and continuous improvement.
Core Principles for Detection Engineering:
- Actionable Alerts: Every alert must allow a human to take a concrete action. If an alert cannot be acted upon, it will be ignored, rendering it worse than useless. Unactionable alerts should be lowered in severity, turned off, or documented with accepted risk signed off by leadership.
- Tightly Focused Rules: Avoid "catch-all" rules. Simpler, more focused rules are easier to tune and provide clearer context for analysts.
- Context is Key: Alerts must provide all necessary context upfront (parent processes, owner, system, HR data, previous alerts) to enable rapid decision-making. This often involves enriching alerts with data from an asset database.
- Strive for Zero False Positives: While never fully achievable, detection engineers should commit to weekly tuning of the "top five noisiest alerts" to continuously reduce noise and improve signal.
Identity as the Perimeter:
- In a distributed workforce, identity is the true perimeter. Firewalls and gateways are no longer sufficient.
- FIDO2 MFA (e.g., biometrics, YubiKeys, Windows Hello) is crucial because it uses certificate pinning, which prevents proxying attacks like Evilginx that can steal session tokens from traditional MFA methods (one-time codes, SMS).
- Session Token Protection: Even with FIDO2, session tokens (cookies) can be stolen. Implement maximum lifetimes, use managed browsers with controlled extensions, and enforce step-up authentication for critical applications (e.g., requiring re-authentication for the SIEM even after SSO login).
Centralized Data and Automation:
- Ingest All Data into the SIEM: This includes security logs, endpoint data, server logs (e.g., Auditbeat for process info), observability data (proxy, netflow, authentication), HR systems, asset databases, and even malware sandbox results.
- Standardize Data with Elastic Common Schema (ECS): Essential for efficient investigations, allowing a single query (e.g.,
source.ip) across diverse data sources. - Cross-Cluster Search: For massive multi-cloud environments (Elastic manages 60+ cloud regions, 500TB/day logs), shipping all logs to a central SIEM is prohibitively expensive ($40,000/day for shipping alone). Instead, logs are kept within regional clusters, and queries are distributed, with results aggregated centrally. This enables searching petabytes of data (e.g., 3 petabytes for Log4Shell) in 30-40 seconds.
- Automate Everything Possible:
- Alert Enrichment: Augment alerts with asset inventory data (owner, contact info, department, time zone), and dynamically add threads with related alerts for the same host/user.
- Alert Distribution: Automatically send alerts to users for validation (e.g., new MFA device added) to close legitimate actions or escalate suspicious ones.
- OSQuery for Forensics: Automatically run OSQuery packs on endpoints upon alert trigger to collect full forensic data (processes, memory, environment variables, installed applications, registry, files) within seconds, crucial for geographically dispersed teams.
- "New Terms" Rules: Detect first-time occurrences of specific events (e.g., an alert that hasn't fired in 30 days) to highlight potentially novel or anomalous activity.
Asset Inventory is Foundational:
- Despite being "boring," a robust asset inventory database is critical. Elastic's model defines "entities" (computers, humans, offices, network zones) and scrapes data daily from sources like GitHub, HR systems, and cloud providers (Azure, GCP, AWS, Active Directory).
- This inventory enables powerful indicator match rules (e.g., "domain administrator" activity, "someone not on my team cloning my repo"—which successfully caught their red team) and enriches alerts with vital context. It also proves invaluable for uncovering Shadow IT.
Detecting Cloud Attacks (Scale, Speed, Complexity):
- Exploitable Services: Beyond internet-facing services, focus on anything handling untrusted data. Detect strange child processes (e.g., Nginx spawning
curlorbash), unusual network connections, and service accounts performing unexpected actions. Jewitt cited catching Kibana zero-days by detecting anomalous child processes. - Misconfigurations: Leverage pre-built detections for common issues (e.g., publicly exposed S3 buckets) and use Cloud Security Posture Management (CSPM) tools (for AWS, GCP, Azure, Kubernetes) to proactively identify risks.
- Compromised Tokens: This is the "biggest fear." Tokens (AWS secret keys, GitHub PATs) are single-factor authentication within the cloud and are easily stolen. Detection is challenging as attackers use them from their own infrastructure. The strategy focuses on detecting token usage from untrusted IP spaces. This involves a sophisticated automated workflow that correlates token usage with known asset inventory, proxy logs, endpoint data, and successful FIDO2 MFA logins to filter legitimate activity.
Technical Deep Dive
▶ Watch: The power of a robust asset inventory for detection (20:30)
Elastic's approach to detection engineering in a multi-cloud, distributed environment is deeply technical, relying on a robust Elastic Stack architecture and sophisticated automation.
At the core of securing the distributed workforce is the emphasis on identity as the perimeter. This is underpinned by the aggressive adoption of FIDO2 Multi-Factor Authentication (MFA). Jewitt highlights that traditional MFA methods like one-time codes or SMS are vulnerable to proxying attacks, such as those facilitated by Evilginx. Evilginx can sit between a user and their legitimate SSO provider, presenting a perfect corporate login page, stealing session tokens, and effectively bypassing MFA. FIDO2, however, uses certificate pinning during the authentication process. When a user authenticates with a biometric (e.g., fingerprint via Windows Hello or macOS) or a physical YubiKey, the client-side certificate is pinned to the company's SSO provider, preventing malicious intermediaries from proxying the session. Despite this, session tokens (browser cookies) remain a target. Elastic mitigates this by setting maximum lifetimes for tokens, using managed browsers with controlled extensions, and implementing step-up authentication for critical applications like their SIEM, requiring a FIDO2 re-authentication even after an initial SSO login.
The foundation for detection is an "all data in the SIEM" strategy. This means ingesting not just security logs but also endpoint data, server logs (using Auditbeat for detailed process information), observability data (CPU, uptime, proxy, netflow, authentication logs), HR system data, asset inventory, and even malware sandbox output. This comprehensive data lake is then standardized using the Elastic Common Schema (ECS), which is crucial for efficient investigations. ECS allows analysts to search for a common field, like source.ip, across all disparate data sources with a single query, significantly reducing triage time.
Given the immense scale—Elastic operates across 60 AWS, Azure, and GCP cloud regions, generating 500 terabytes of logs daily—shipping all this data to a single central SIEM is financially impractical, incurring an estimated $40,000 per day in data transfer costs alone. To address this, Elastic employs cross-cluster search. Logs are retained within regional Elasticsearch clusters, often categorized as "observability logs" to leverage existing budgets. The central SIEM cluster then distributes queries across these regional clusters, aggregating the results. This architecture proved its value during Log4Shell, allowing searches for JNDI strings across approximately 3 petabytes of hot data within 30-40 seconds.
Automation is a cornerstone of Elastic's detection strategy. Alert enrichment is heavily utilized. When an alert fires, Elastic's automation system queries the asset inventory to pull in details like the system owner, their Slack handle, phone number, department, and even time zone from HR records (Workday). It also queries the SIEM for all other alerts related to that hostname and username over the past seven days, embedding these as threads in the Slack alert. This provides immediate context, enabling rapid decision-making, even from a mobile phone. Another powerful automation is the integration of OSQuery. Upon a malware alert, a pre-configured OSQuery pack is automatically executed on the affected workstation, collecting a full forensic snapshot (running processes, memory information, environment variables, installed applications, registry data, files) within seconds. This data is then attached to the alert, providing crucial evidence for responders who might be waking up hours later in a different time zone. Elastic also uses "new terms" rules, a feature that flags events that haven't been seen in a defined period (e.g., 30 days), elevating their priority even if they would normally be low-severity.
The asset inventory is not just a database; it's a dynamic data model. Elastic defines "entities" (computers, human beings, office buildings, network zones) and daily scrapes data from diverse sources like GitHub, HR systems, and all cloud providers. This normalized data is ingested into a single Elasticsearch index. This inventory enables sophisticated indicator match rules, such as identifying when a non-security team member attempts to clone a security team's private Git repository—a rule that famously caught Elastic's red team during an engagement. This inventory also forms the basis for rich alert contextualization and the identification of Shadow IT.
Cloud-specific detection strategies tackle the unique challenges of scale, speed, and complexity:
- Exploitable Services: For detecting zero-days or known vulnerabilities, the focus is on anomalous process execution. Jewitt provided an example of detecting Kibana zero-days by observing Kibana spawning unexpected child processes. The methodology involves baselining normal process behavior (e.g., Nginx spawning itself for load balancing, or known upgrade processes) and then creating rules that alert on any deviation (e.g., Nginx spawning
curlorbash). Detecting unusualwgetorcurlprocesses, although noisy, is crucial given their common use by attackers for command and control. - Misconfigurations: Leveraging readily available tools and detections is key. Elastic uses pre-built rules for common misconfigurations (e.g., publicly exposed S3 buckets) and employs Cloud Security Posture Management (CSPM) tools to provide management with dashboards of risks across AWS, GCP, Azure, and Kubernetes environments.
- Compromised Tokens: This is the most challenging area. Cloud tokens (AWS secret keys, GitHub Personal Access Tokens (PATs)) are effectively single-factor authentication within the cloud ecosystem, often stored as clear-text strings. Attackers can steal these and use them from their own infrastructure, bypassing endpoint and proxy logs. Elastic's solution is an automated workflow, often orchestrated via a SOAR platform like Tines.
- Trigger: A "new terms" rule detects
git cloneorgit fetchactivity from a private repository originating from an IP address not seen in the last 30 days. - SOAR Workflow: A webhook sends the full alert (as JSON) to Tines.
- Validation Chain: Tines executes a series of HTTP requests to validate the activity:
- Is the source IP in Elastic's asset database?
- Is the source IP seen in EDR (endpoint) or proxy logs (managed workstation)?
- Is the source IP associated with known serverless automation (e.g., GitHub Actions, AWS Lambda) via Filebeat logs?
- Was there a successful FIDO2 MFA login from that IP in Okta logs within the last 24 hours?
- Decision & Escalation: If any validation query returns a positive match, the alert is automatically closed in the SIEM, improving metrics. If, after a configurable delay (e.g., 5 minutes), the alert is still open, it is sent to Slack with full enrichment. If it remains unacknowledged after an SLA-defined period (e.g., 2 hours), it escalates to PagerDuty. This workflow is highly effective at identifying Shadow IT initially, then focusing on truly anomalous token usage.
This intricate web of data ingestion, standardization, distributed querying, and intelligent automation illustrates Elastic's sophisticated approach to detection engineering, turning overwhelming complexity into actionable security intelligence.
Demo / Proof of Concept
▶ Watch: Cloud challenges: immense scale, speed, and inherent complexity (26:00)
While Aaron Jewitt's talk did not feature a live, interactive demonstration of a tool or exploit, he effectively presented a "proof of concept" through detailed walkthroughs of Elastic's internally developed detection engineering workflows and specific examples of their efficacy.
A key "demonstration" was the detailed explanation of the compromised token detection workflow. Jewitt meticulously outlined the multi-step, automated process (often orchestrated via Tines) that Elastic uses to validate git clone or git fetch activities from private repositories originating from previously unseen IP addresses. He walked through the logical sequence: from the initial "new terms" rule trigger, through a series of automated HTTP requests querying the asset database, EDR/proxy logs, Filebeat logs for serverless automation, and Okta for FIDO2 MFA logins. This detailed explanation served as a powerful conceptual proof of how complex, multi-source correlation can be automated to filter legitimate activity from genuine threats, drastically reducing manual triage.
Furthermore, Jewitt provided concrete examples of successful detections:
- Catching the Red Team: He shared an anecdote where a custom indicator match rule – "alert me if anyone not on my team clones my repo" – successfully identified their internal red team during an engagement. This highlighted the power of leveraging the asset inventory for highly specific and effective detections.
- Detecting Zero-Days: Jewitt mentioned detecting zero-day vulnerabilities in Kibana (Elastic's own product) by observing anomalous child processes being spawned by the Kibana service. This validated the methodology of baselining normal process behavior and alerting on deviations.
- Log4Shell Incident: The successful use of cross-cluster search to rapidly query 3 petabytes of data for JNDI strings within 30-40 seconds during the Log4Shell crisis served as a real-world proof of concept for the architecture's speed and scale. He even recounted an instance where an attacker exploited an AWS backend service via Log4Shell, which Elastic detected and reported to AWS within minutes, showcasing the speed of their detection capabilities even for third-party vulnerabilities.
These examples, woven throughout the technical deep dive, served as compelling evidence of the practical success and real-world applicability of the detection engineering principles and automated systems Elastic has implemented.
Defensive Implications
▶ Watch: Q&A: Log storage strategy and ML for human detection (48:30)
Aaron Jewitt's talk offers profound defensive implications for any organization navigating the complexities of a distributed workforce and multi-cloud environment. The core message is a call to action for a paradigm shift in security operations.
First and foremost, defenders must fully embrace identity as the new perimeter. This means prioritizing and rigorously implementing strong, phishing-resistant FIDO2 MFA across the entire organization, leveraging biometrics, YubiKeys, or Windows Hello to defeat sophisticated proxying attacks like Evilginx. Beyond authentication, protecting session tokens is paramount; organizations should enforce maximum token lifetimes, mandate managed browsers with controlled extensions, and implement step-up authentication for access to critical systems.
Secondly, centralizing all relevant data in the SIEM is non-negotiable. This extends beyond traditional security logs to include endpoint telemetry, server audit logs (Auditbeat), comprehensive observability data (proxy, netflow, authentication), HR system data, and a robust asset inventory. Standardizing this vast dataset with a schema like Elastic Common Schema (ECS) is crucial for enabling rapid, cross-source investigations. For multi-cloud operations, adopting a cross-cluster search architecture is vital to manage immense log volumes (e.g., 500 TB/day) cost-effectively, avoiding exorbitant data transfer fees while maintaining query speed.
Thirdly, aggressive automation is no longer a luxury but a necessity. Security teams must invest in SOAR platforms (like Tines) to automate alert enrichment, automatically pulling in critical context from asset inventories and HR systems, and threading related alerts. Implementing OSQuery for automated, real-time forensic data collection on endpoints upon alert trigger can drastically reduce response times for geographically dispersed teams. Automation should also extend to alert distribution, enabling self-service validation for legitimate activities and automatic case creation and escalation for confirmed threats. The use of "new terms" rules can help defenders proactively identify novel or anomalous activities that might otherwise be missed by static signatures.
Fourthly, a meticulously maintained asset inventory is the bedrock of effective detection engineering. Organizations must invest the effort to build and continuously update a comprehensive database of all entities (users, systems, cloud assets, network zones). This inventory is indispensable for enriching alerts, creating highly targeted indicator match rules (e.g., detecting unauthorized access to sensitive repositories), and crucially, for uncovering Shadow IT—systems that pose significant unmanaged risk.
Finally, proactive cloud security requires tailored strategies. For exploitable services, defenders should baseline normal process execution for critical applications (e.g., web servers) and implement rules that alert on any anomalous child processes or unusual network connections. For misconfigurations, widespread adoption of Cloud Security Posture Management (CSPM) tools and leveraging pre-built detections for common cloud misconfigurations (e.g., publicly exposed S3 buckets) is essential. The most challenging threat, compromised tokens, demands sophisticated, multi-source detection workflows. Defenders must correlate token usage with known legitimate IP spaces, managed endpoints, and recent FIDO2 MFA authentications. Furthermore, continuous monitoring for leaked tokens in public repositories or Pastebin (e.g., using tools like TruffleHog) is a critical preventative measure.
Ultimately, detection engineering is a "never-ending story" of continuous improvement. Defenders must allocate dedicated time weekly to tune noisy alerts, adapt to new technologies, and regularly reassess their fears and attack scenarios to stay ahead of evolving threats.
Key Takeaways
- Identity is the Perimeter: In distributed, multi-cloud environments, strong FIDO2 MFA (biometrics, YubiKeys) with certificate pinning is critical to defeat proxying attacks, and protecting session tokens through strict policies and step-up authentication is essential.
- Centralize All Data (and Standardize): Ingest all relevant data (security, observability, HR, asset inventory) into your SIEM, using a common schema like Elastic Common Schema (ECS), to enable rapid and comprehensive investigations.
- Leverage Cross-Cluster Search for Scale: For large multi-cloud deployments, storing logs regionally and using cross-cluster search is a cost-effective and performant way to query petabytes of data without incurring massive data transfer costs.
- Automate Everything Possible: Prioritize alert enrichment (context from asset inventory, HR, past alerts), alert distribution (user validation, auto-case creation), and automated forensics (e.g., OSQuery integration) to improve speed and reduce analyst burnout.
- Asset Inventory is Foundational: A robust, constantly updated asset inventory is indispensable for providing context to alerts, enabling powerful indicator match rules, and uncovering Shadow IT.
- Target Cloud-Specific Threats: Develop tailored detection strategies for exploitable services (anomalous child processes), misconfigurations (CSPM tools), and especially compromised tokens (sophisticated correlation of token usage with trusted IPs and authentication data).
About the Speaker(s)
Aaron Jewitt is the Detection Engineering Team Lead at Elastic. With over 20 years of experience in cybersecurity, he brings a wealth of knowledge from diverse roles. He began his career at the National Security Agency (NSA), spending 12 years on what he refers to as the "red team side" at TAO (Tailored Access Operations). For the past seven years, he has focused on threat hunting and detection engineering, with the last four years dedicated to protecting Elastic's globally distributed workforce as part of their internal infosec team. His background provides him with a unique perspective on both offensive and defensive security, which he applies to building pragmatic and effective detection capabilities.
Reviews
Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT
Cloud security talks are usually zero-trust marketing bullshit, but this is a concrete blueprint from an ex-TAO operator on how to actually do detection engineering at scale. Jewitt details the exact Tines workflows, OSQuery automation, and cross-cluster architectures Elastic uses to survive without going bankrupt on data ingest fees. If you run a SOC in the cloud, you need to watch this.
Heather Calloway (CISO) — SOLID
This is a blueprint for operating a modern, cloud-native security program without drowning in noise or cloud egress costs. It successfully moves detection engineering from an academic exercise into a risk-managed, financially literate business function.