Scaling Detection and Response Teams - Enabling Efficient Investigations

James Dorgan (Principal Incident Responder · Coinbase)

Security BSides London 2023 · Day 1 · Track 2

Overview

In his compelling BSides London talk, "Scaling Detection and Response Teams - Enabling Efficient Investigations," James Dorgan, a Principal Incident Responder at Coinbase, tackles a pervasive challenge facing modern security operations centers (SOCs): the struggle to scale incident response capabilities in the face of ever-increasing data volumes and alert fatigue. Dorgan argues that while significant advancements have been made in logging and detection, the crucial phase of investigation often remains woefully inefficient, relying on manual processes that hinder a team's ability to effectively detect and respond to threats.

Watch on YouTube

Visual summary for Scaling Detection and Response Teams - Enabling Efficient Investigations by James Dorgan
Visual summary for Scaling Detection and Response Teams - Enabling Efficient Investigations by James Dorgan

Key moments

  1. 0:00 Introduction and talk scope
  2. 8:00 Symptoms of limited investigation maturity
  3. 10:40 Case Study 1: Automating investigation workflows
  4. 15:00 Demo: Centralized investigation application
  5. 24:00 Case Study 2: Building context into detections
  6. 29:50 Case Study 3: Bringing employees into triage
  7. 35:00 Key takeaways for scaling teams effectively
  8. 38:40 Q&A: Challenges of building and securing custom tools

Scaling Detection and Response Teams - Enabling Efficient Investigations

Speakers: James Dorgan, Principal Incident Responder, Coinbase

Conference: BSides London

YouTube: https://www.youtube.com/watch?v=sHDJAb1rY2c

Overview

In his compelling BSides London talk, "Scaling Detection and Response Teams - Enabling Efficient Investigations," James Dorgan, a Principal Incident Responder at Coinbase, tackles a pervasive challenge facing modern security operations centers (SOCs): the struggle to scale incident response capabilities in the face of ever-increasing data volumes and alert fatigue. Dorgan argues that while significant advancements have been made in logging and detection, the crucial phase of investigation often remains woefully inefficient, relying on manual processes that hinder a team's ability to effectively detect and respond to threats.

Dorgan’s presentation provides a practical, experience-driven roadmap for organizations seeking to enhance their investigative maturity. He highlights how the current paradigm, which often pushes raw, uncontextualized alerts to human analysts, leads to delayed triage, inconsistent investigations, and debilitating analyst fatigue. Through three innovative case studies implemented at Coinbase, Dorgan demonstrates how automation, intelligent data enrichment, and strategic employee engagement can fundamentally transform the investigation process, freeing up analysts to focus on deeper, more meaningful threat analysis rather than repetitive data correlation. This talk is particularly relevant for mature organizations aiming to optimize their security operations and achieve truly scalable detection and response capabilities.

Background

▶ Watch: Introduction and talk scope (0:00)

The journey of a security alert, from log ingestion to final response, can be conceptualized through frameworks like Jared Atkinson's "Funnel of Fidelity," which outlines key stages: logging, detection, investigation, and response. Dorgan observes that while logging and detection have seen substantial industry-wide maturity, often supported by sophisticated Security Information and Event Management (SIEM) solutions like Splunk, Elastic Search, Snowflake, and LogRhythm, or open-source projects like Sigma for detection rules, the investigation phase frequently lags. Many organizations possess vast quantities of security data but lack efficient mechanisms for analysts to rapidly extract meaningful insights from it.

The core problem Dorgan identifies is a reliance on human analysts to perform extensive manual data correlation at the point of an alert. When a detection fires, such as an "anomalous login from an IP address geolocated in France," the analyst is often presented with minimal information—perhaps just the alert title in a ticketing system like Jira. They are then expected to embark on a lengthy, multi-data source investigation path. This can involve querying identity provider logs, 2FA logs, HR tools, asset management systems, EDR consoles, and even internal communication platforms like Slack, all while cross-referencing past alerts. Each query requires specific syntax, time range adjustments, and error correction, consuming valuable minutes per question.

Dorgan illustrates this with the "anomalous login" example. An analyst would need to ask: What are the user's typical login patterns? Is the detection logic flawed? What team does the user work in, and are they expected to be in France? What devices are assigned to them (e.g., Windows vs. MacBook)? Are they online and reachable via Slack? What other alerts exist for this user or their devices? Even with all data centralized in a SIEM, manually running these queries can take 20 minutes just to establish a baseline understanding—a critical delay if a threat actor is involved. This manual burden leads to several detrimental symptoms: delayed alert triaging, where initial understanding is slow; a lack of shared investigative baseline, meaning different analysts (e.g., a junior vs. senior) will follow inconsistent paths, potentially leading to varied or even incorrect resolutions; and pervasive analyst fatigue, as repetitive, low-value data gathering drains focus and concentration before deep analysis even begins. Ultimately, the time taken per detection becomes unsustainable, directly hindering a team's ability to scale with an increasing volume of high-fidelity alerts.

Key Findings

▶ Watch: Case Study 1: Automating investigation workflows (10:40)

The central finding of Dorgan's talk is that the traditional, largely manual approach to security investigations is fundamentally unscalable and detrimental to the efficiency and morale of detection and response teams. By focusing on the investigation phase, which represents a high-volume daily activity compared to the rarer "response" phase, organizations can achieve significant returns on investment in terms of scaling their capabilities.

Coinbase's experience, as shared by Dorgan, demonstrates that this problem can be systematically addressed through a multi-pronged strategy:

  1. Automating Investigation and Response Workflows: By building a centralized application that acts as an "investigation co-pilot," teams can abstract away the complexity of querying disparate data sources and executing response actions. This shifts the burden from manual, repetitive tasks to a streamlined, automated process, significantly reducing the time to establish a baseline understanding of an alert.
  2. Building Context into Detections: Proactively enriching raw detection data with relevant user and machine profile information eliminates the need for analysts to manually correlate basic facts. By leveraging continuously updated context tables, every alert can arrive with immediate, critical information about the affected entities, transforming ambiguous alerts into actionable insights.
  3. Bringing Employees into the Triage Process: For certain types of alerts, especially those related to user behavior, engaging the end-user directly via automated communication channels (like a Slack bot) can offload initial triage work from security analysts. This crowdsourcing of information, when treated as a data source rather than an absolute authority, can resolve a significant number of detections without requiring analyst intervention.

Collectively, these approaches lead to a substantial reduction in alert triaging time, foster a shared investigative baseline across the team, and mitigate analyst fatigue. Dorgan estimates that these efforts save Coinbase's analysts 30-50 hours per month, allowing them to focus on complex threat hunting and deep incident analysis, thereby enabling the team to scale effectively.

Technical Deep Dive

▶ Watch: Case Study 2: Building context into detections (24:00)

Dorgan’s presentation delves into the technical specifics of Coinbase’s solutions, highlighting how they engineered efficiency into their incident response workflows.

Automating Investigation and Response Workflows (Case Study 1)

The cornerstone of Coinbase's approach is a centralized application designed to automate the initial investigation and response steps. This application is built around the concept of a "starting point" – a single piece of information, such as an email address, serial number, IP address, hash, EDR GUID, or Cloud ARN, from which an investigation can begin.

The application’s architecture is designed to perform three core functions:

  1. Historical Log Querying: Upon receiving a starting point (e.g., an email address), the application automatically queries various historical data sources. This includes Single Sign-On (SSO) logs, asset inventory logs, and SaaS application logs, retrieving all relevant information associated with that entity. This replaces the manual process of analysts writing complex queries across different SIEMs or log aggregators.
  2. Live Data API Integration: Beyond historical data, the application connects to various security and IT systems via APIs to pull real-time information. This allows analysts to instantly check a user’s account suspension status, an endpoint’s isolation status, or other live attributes without navigating to separate consoles.
  3. Automated Response Tasks: The application integrates with various security tools to execute predefined response actions. Instead of manually logging into an EDR console or an SSO platform to suspend an account, invalidate 2FA tokens, or force a password change, analysts can initiate these actions directly from the central application. This not only speeds up response but also ensures consistency in execution.

From an architectural standpoint, Dorgan revealed in the Q&A that this application evolved from a "Christmas project" into a robust tool. It leveraged existing building blocks or frameworks within Coinbase's engineering ecosystem, specifically a platform for rapidly building web applications and another for creating logic blocks and workflows. Under the hood, it primarily uses SQL queries to interact with data already residing in their SIEM and curl commands for API calls to various upstream systems. This approach allowed analysts, rather than dedicated software engineers, to rapidly develop and iterate on solutions to their immediate problems.

The benefits of this abstraction and automation are manifold:

  • Crowdsourced Best Practices: Experienced analysts can contribute and refine the "best-in-class" queries and workflows within the application, making their expertise immediately available to the entire team. This eliminates the siloed knowledge often held in personal notes or Confluence pages.
  • Rapid Onboarding: New analysts, who previously took months to learn all the data sources and queries, can now become productive in weeks by simply leveraging the pre-built workflows.
  • Platform Agnostic Response: The application abstracts underlying security technologies. For instance, an "isolate endpoint" button can seamlessly interact with different EDR platforms (e.g., during a migration from one EDR vendor to another) without the analyst needing to know which EDR is installed on a specific machine.

Building Context into Detections (Case Study 2)

The second case study focuses on enriching raw detection data before it reaches the analyst, thereby eliminating much of the initial manual lookup. This is achieved through the creation and maintenance of two critical context tables:

  1. User Profile Table: For every employee, this table consolidates key data from various sources: SSO ID, login email, location, organizational unit (org), job role, assigned TFA devices, assigned endpoints, and even trusted IP addresses strongly associated with that user (e.g., from U2F security key logins).
  2. Machine Profile Table: For every endpoint device, this table tracks its serial number, owner, platform type (e.g., Windows, macOS), installed EDR, EDR ID, online status, whether it's been erased, known IP addresses, and connected USB devices.

A dedicated system runs on a schedule, querying upstream data sources (SSO system, HR/Org system, TFA provider, EDR) and consolidating this information into these two tables. The refresh interval varies based on data freshness requirements; for example, a job role might update daily, while IP addresses or USB device connections are updated more frequently.

These tables serve as crucial pivot points for investigations. If a detection comes in with only a serial number, an analyst would historically have to manually look up that serial number in an asset management tool, find the owner, and then search for information about that owner. With the context tables, a lookup against the Machine Profile table for the serial number immediately provides details about the device (platform, EDR, USB devices) and, critically, links to the owner's User Profile, revealing their location, role, TFA devices, etc. This transforms an ambiguous alert into a rich, contextualized incident.

A notable application of this approach is in refining 2FA enrollment detections. Initial detections for new 2FA device enrollments often generate a high volume of false positives in large organizations, especially when users are issued multiple backup keys. By joining the 2FA enrollment event with the Machine Profile table, Coinbase can check if the IP address from which the enrollment occurred is one of the historical IPs associated with any of the user's corporate devices. If it is, the alert is deprioritized or suppressed, as it's likely a legitimate enrollment from a trusted corporate device. If the IP is not associated with a corporate device, it becomes a higher-fidelity alert, indicating potential enrollment from a personal device (a policy violation) or, more critically, by a threat actor. This method converted a previously unusable detection into an effective security control.

Demo / Proof of Concept

▶ Watch: Case Study 3: Bringing employees into triage (29:50)

Dorgan provided a compelling demonstration of the centralized investigation and response application, showcasing its real-world utility at Coinbase. The demo highlighted how the application rapidly consolidates information that would otherwise require significant manual effort.

Investigation Workflow Demo:

The demonstration began by inputting a single investigation fact – in this case, James Dorgan's email address – into the user investigation tool. The application then rapidly queried various data sources and APIs to present a comprehensive profile:

  • User Status: Immediate display of account activity (e.g., "active," "not suspended"), TFA status, messenger activity, account creation date, and last password change.
  • Organizational Context: Details such as the user's team, organizational unit, hire date, manager, employment status (full-time/contractor), job title, and physical location. Crucially, it also highlighted if the user had any risky permissions assigned, which would alter the response strategy in a compromise scenario.
  • Asset Management: A list of all devices assigned to the user, including serial number, hostname, EDR ID, EDR type, platform, last online status, and critical real-time status indicators (e.g., "online," "contained," "locked," "in process of being erased"). This allows for quick, light-touch response actions directly from the user's profile, such as suspending an account or forcing a password change.
  • Behavioral Workflows: The demo then navigated to other workflows:
  • Login Activity: A comprehensive list of SSO logins, showing the application accessed (redacted in the demo), IP address, country, city, user agent, and ISP. This helps identify anomalous login patterns.
  • Visualizations: Geographical representation of login activity, showing typical login locations (e.g., United Kingdom for Dorgan) versus anomalous ones (e.g., a few logins from the United States).
  • TFA Activity: Details on how TFA was satisfied (e.g., security key vs. bypass code), highlighting potential red flags like bypass code usage.
  • Related Detections and Incidents: The application integrates with Coinbase's incident and ticketing system, pulling all past and current alerts and incidents related to the user or their devices. For example, it showed three incidents related to Dorgan's user, including one for remote access tooling (RustDesk) being installed, which was hypothetically "incorrectly resolved as a false positive." This provides crucial historical context for current investigations.

The entire process, which Dorgan previously estimated would take 20+ minutes of manual querying, was completed in seconds, providing a comprehensive baseline understanding for the analyst.

Response Workflow Demo:

The second part of the demo illustrated the automated response capabilities:

  • Response Catalog: The application maintains a catalog of available response actions, allowing analysts to self-discover capabilities like "retrieving remote files," "retrieving browser histories," or "suspending user accounts."
  • Detailed Descriptions: Each response task includes a clear description of what it does. For instance, suspending a user account would not only invalidate their SSO account but also their 2FA account, workspace account, invalidate tokens, and delete backup codes, providing a holistic understanding of the impact.
  • Incident Integration: Response tasks can be tied to an active incident or investigation. When a task is run, the application automatically logs who, what, where, why, and when the task was executed into the associated incident ticket and sends a message to the incident Slack channel, ensuring full visibility for the team.
  • Target Selection & Context: Analysts can select one or multiple users for a response task (e.g., "suspend user" and "force user sign out"). They are then prompted to provide context for the action, which is also logged.
  • Task History & Artifacts: All executed response tasks are tracked, showing the author, target, and outcome. Critically, if a task retrieves an artifact (e.g., a browser history file), it is stored in a central, accessible location, preventing duplicate efforts by other analysts.

Dorgan emphasized the significant impact of this system, estimating it saves 30-50 hours of analyst time per month, freeing them to focus on more complex and strategic security work.

Defensive Implications

▶ Watch: Q&A: Challenges of building and securing custom tools (38:40)

The strategies outlined by James Dorgan offer critical defensive implications for security teams striving for greater efficiency and scalability:

  1. Prioritize Investigation Maturity: Organizations must recognize that advanced logging and detection are only part of the solution. Investing in the investigation phase is crucial for scaling. This means moving beyond simply pushing raw alerts to analysts and instead focusing on streamlining the entire investigative workflow.
  2. Adopt or Build Centralized Automation Platforms: Defenders should explore implementing Security Orchestration, Automation, and Response (SOAR) platforms or developing internal tools that centralize common investigation and response tasks. These platforms should be capable of:
  • Taking a single indicator of compromise (IOC) or entity (e.g., IP, username, serial number) as a starting point.
  • Automatically querying multiple disparate data sources (SIEM, EDR, HR, Asset Management, SSO, Cloud APIs) to gather baseline context.
  • Executing predefined response actions (e.g., account suspension, endpoint isolation, password reset) without requiring manual interaction with multiple consoles.
  • Abstracting underlying technology differences (e.g., supporting multiple EDR vendors seamlessly).
  1. Establish Robust Contextual Data Tables: Creating and continuously updating User Profile and Machine Profile tables is a foundational step. These tables, populated from authoritative sources (HR systems, SSO providers, EDRs, asset management tools), provide a rich, real-time context for every entity within the environment. This enables:
  • Automated Alert Enrichment: Detections can be automatically enriched with details like an asset's owner, their department, location, and associated devices, eliminating manual lookups.
  • Advanced Detection Logic: As demonstrated with the 2FA enrollment detection, contextual data allows for more sophisticated and higher-fidelity detection rules that filter out noise by correlating events with known trusted behaviors or entities.
  1. Leverage End-Users for Initial Triage (with Caution): For appropriate, lower-risk, and high-volume alerts (e.g., anomalous login from an unfamiliar location), integrate Slack bots or similar communication tools to directly engage the affected user for initial triage. This offloads significant workload from analysts. However, it is paramount to:
  • Treat user responses as a data source, not an authority. Implement mechanisms like 2FA verification for "I recognize this" responses.
  • Have clear escalation paths for "I don't recognize this" responses, potentially triggering immediate automated response actions or prioritizing analyst review.
  • Be mindful of insider threat or social engineering scenarios where user verification might be compromised.
  1. Standardize Investigation Baselines: Ensure all analysts, regardless of experience level, follow a consistent, documented process for investigating common alert types. The automated platform can enforce this by guiding analysts through predefined workflows and presenting standardized information. This reduces variability in investigation quality and resolution.
  2. Focus on "Stop the Bleed" Automation: Automate immediate containment actions for high-severity threats. If an analyst determines an alert is a true positive, the system should allow them to rapidly suspend accounts, isolate endpoints, or kill sessions with minimal clicks, enabling them to then conduct a more methodical investigation.
  3. Measure and Iterate: Continuously track the time saved, the consistency of investigations, and the reduction in analyst fatigue. Use these metrics to identify new areas for automation and improvement, fostering a culture of continuous improvement in security operations.

By implementing these defensive strategies, organizations can transform their SOCs from reactive, manually intensive operations into proactive, scalable, and highly efficient threat response centers.

Key Takeaways

  • Centralize and Automate Investigation Tasks: Identify repetitive, manual queries and response actions that analysts perform daily. Implement a centralized platform or SOAR solution to automate these workflows, significantly reducing the time spent on baseline data gathering and initial response, thereby saving 30-50 hours per month of analyst time.
  • Build Robust Context Tables: Develop and maintain dynamic User Profile and Machine Profile tables by consolidating data from all authoritative sources (HR, SSO, EDR, asset management). These tables serve as crucial pivot points, enriching raw alerts with immediate, comprehensive context about affected entities.
  • Enrich Detections at Source: Avoid sending analysts raw, uncontextualized alerts (e.g., just a serial number). Pre-populate detections with key investigative facts such as the owner, their team, location, device type, and recent activity, reducing the need for manual lookups.
  • Leverage Employees for Initial Triage: For suitable high-volume, lower-risk alerts (e.g., anomalous logins), implement automated systems (like Slack bots) to engage end-users for initial verification. Treat their responses as a valuable data source for triage, but not as an absolute authority, and ensure robust escalation paths.
  • Establish a Shared Investigative Baseline: Use automated workflows and enriched detections to ensure all analysts, regardless of experience, follow consistent investigative paths and access the same high-quality, pre-correlated information. This improves consistency, reduces analyst fatigue, and accelerates onboarding for new team members.
  • Focus on Continuous Improvement: Regularly assess existing processes, identify new opportunities for automation, and iterate on solutions. The goal is to continuously free up analysts from mundane tasks, allowing them to focus their energy and expertise on deeper threat analysis, hunting, and complex incident response, thus enabling the team to scale effectively.

About the Speaker(s)

James Dorgan is a Principal Incident Responder within the Security Incident Response Team (SIRT) at Coinbase. With over a decade of experience in the cybersecurity field, James has held various roles spanning incident response, Managed Detection and Response (MDR), and security research. He began his career as a SysAdmin before transitioning into security. His professional interests lie in incident response, detection engineering, and continuous improvement, which involves optimizing daily processes for detection and response teams to enhance overall capability and scalability. James has a long-standing connection with BSides London, having attended the conference for 10 years, with his first cybersecurity conference being BSides London in 2012.

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

This is a meat-and-potatoes defensive engineering talk. Dorgan isn't dropping 0-days; he's showing how Coinbase dug their SOC out of alert fatigue by building their own internal SOAR. If your analysts are still manually querying Splunk and cross-referencing Workday just to figure out if a French IP address actually belongs to a deployed laptop, watch this.

Heather Calloway (CISO) — STRONG ACCEPT

If your SOC manager is asking for more headcount to handle alert volume, have them watch this first. Coinbase demonstrates how shifting engineering cycles toward the investigation phase yields measurable ROI in analyst hours and retention. It’s a practical blueprint for stopping the bleed on tier-one operational budgets.

→ Top-rated talks at Security BSides London 2023

All talks from Security BSides London 2023