Automated Incident Response

Automated Incident Response

Speakers: Valentine Mairet (GitLab), Harjeet Sharma (GitLab)

Conference: HITB Phuket

YouTube: https://www.youtube.com/watch?v=1h5U3VwlWa8

Overview

In this comprehensive talk and workshop at HITB Phuket, Valentine Mairet and Harjeet Sharma from GitLab's Security Incident Response Team (SIRT) unveiled their sophisticated approach to Automated Incident Response. The session, titled "Automated Incident Response," provided both a theoretical foundation and a practical, hands-on experience, demonstrating how GitLab tackles the pervasive challenges of modern security operations through extensive automation. The core of their presentation focused on the "Guard" system, GitLab's internally developed framework for orchestrating detection, incident management, and metric generation across multiple platforms, notably Tines (a SOAR platform), GitLab itself, Slack for communication, and Dvo as their SIEM.

The speakers highlighted that incident response (IR) extends far beyond mere investigation, encompassing detection engineering, rigorous administrative tasks, and continuous process improvement. They articulated a compelling vision where automation isn't just about speed but also about standardization, consistency, and enabling IR teams to scale effectively despite being perpetually understaffed and battling alert fatigue. By showcasing their real-world implementations, Mairet and Sharma provided a blueprint for other organizations seeking to elevate their security posture through intelligent, integrated automation that transforms reactive processes into proactive, data-driven operations.

Background

▶ Watch: Introduction and Workshop Agenda Overview (0:00)

The GitLab SIRT operates as a global, 24/7 team, split across APac, Amia, and Americas regions. This geographical distribution, while ensuring continuous service, introduces significant administrative overhead, especially concerning time zone management and incident handovers. The team's daily operations involve investigating incidents reported by team members, triaging security alerts from their Dvo SIEM, and collaborating extensively on the GitLab platform for documentation and incident tracking, while using Slack for real-time communication. This constant context-switching between platforms and the sheer volume of tasks underscored the critical need for automation.

The speakers identified three primary areas ripe for automation within their IR lifecycle:

1. Detection Engineering: This phase traditionally involves manual review, deployment of alerts to the SIEM, and cumbersome metric generation. GitLab aimed to automate the deployment of alerts to their SIEM, ensure high-quality reviews, generate crucial metrics to justify team growth and tool investments, and build a SIEM-agnostic pipeline to facilitate future platform changes without rework. The goal was to standardize the format and content of detections, ensuring consistent quality.
2. Incident Management: Beyond the "cool stuff" of forensics, effective incident management demands streamlined reporting, initial triage, and robust documentation. The team sought to automate the easy reporting of incidents, establish automated initial severity and priority triage, keep documentation up-to-date using GitLab as a single source of truth, facilitate smooth global handovers, ensure high-quality and consistent incident reviews, and generate metrics about incidents to inform strategic decisions.
3. Metrics Generation: A recurring theme was the necessity of data to drive improvements. Automating metric generation provides insights into alert noise, identifies trends, helps reduce alert fatigue by pinpointing false positives, and ultimately justifies headcount and tool acquisitions to leadership. Engineers often suffer from alert fatigue, and objective metrics provide the evidence needed to tune alerts effectively.

The overarching motivations for GitLab's aggressive pursuit of automation were clear:

• Efficiency: Streamlining tasks to save time and resources.
• Standardization: Ensuring consistent processes and outputs across the global team.
• Time Saving: Eliminating manual, repetitive tasks like revoking leaked API keys.
• Do Not Forget: Implementing automated reminders and workflows to prevent missed critical tasks.
• Scalability: Enabling the team to handle an increasing volume of alerts and incidents without a proportional increase in manual effort. These principles guided the development of GitLab's comprehensive automation framework.

Key Findings

▶ Watch: Key Automation Opportunities in Incident Response (8:10)

GitLab's journey into Automated Incident Response has yielded several significant contributions and findings, consolidating their approach into a powerful and integrated system. The most prominent outcome is the development and implementation of Guard (GitLab Universal Automated Response and Detection), an internal "beast" that orchestrates their entire alerting pipeline and Slack handling capabilities. Guard embodies their philosophy of comprehensive automation, covering everything from alert creation and deployment to tuning, and encompassing all aspects of incident management, including reporting, handling, handovers, and reviews, while continuously generating vital metrics.

A key component of Guard is the User Attestation Module (UAM), an innovative automation designed to reduce false positives and streamline initial alert triage. By directly engaging users via Slack to validate suspicious activities (e.g., the creation of a personal access token (PAT)), UAM significantly offloads the initial investigative burden from security engineers, allowing them to focus on confirmed threats. This module exemplifies how proactive user interaction can enhance IR efficiency.

Furthermore, GitLab has firmly established the GitLab platform itself as the single source of truth for all incident-related information. By leveraging GitLab issues as incident records, they centralize documentation, timelines, and communication, ensuring consistency and ease of knowledge transfer, especially crucial for a global team. This integration of their core product into their security operations workflow not only demonstrates internal dogfooding but also provides a powerful, version-controlled environment for incident management. The extensive use of Tines as their central SOAR platform was also a critical finding, demonstrating how a no-code/low-code solution can effectively integrate disparate security tools and automate complex workflows across SIEMs, communication channels, and incident management systems. The commitment to detection as code—managing detection rules in a GitLab repository with a rigorous merge request (MR) workflow—ensures high-quality, standardized, and auditable alert deployment, a fundamental shift from traditional manual SIEM rule management.

Technical Deep Dive

▶ Watch: Demo: User Attestation Module (UAM) for Triage (21:50)

GitLab's Automated Incident Response framework is a sophisticated orchestration of several platforms, with Tines serving as the central SOAR platform that glues everything together. The entire system is designed for efficiency, standardization, and scalability, addressing common pain points in security operations.

Detection as Code and Automated Deployment

The foundation of GitLab's detection strategy is detection as code. All detection rules are managed within a dedicated GitLab repository. When a security engineer develops a new detection rule, they create a merge request (MR). This MR triggers an automated pipeline that performs various quality checks, ensuring the rule adheres to standardized formats and content guidelines. This process ensures high-quality, consistent detections.

After successful pipeline checks, the MR is assigned to another security engineer for peer review and approval, enforcing a collaborative and robust quality assurance process. Automated notifications are sent to relevant Slack channels for new MRs, prompting timely reviews. Once approved and merged, another pipeline is automatically triggered. This pipeline fetches the changed files and uses the Dvo SIEM API to deploy the new or updated detection rule directly to the SIEM. This entire process is designed to be SIEM-agnostic, meaning the underlying logic can be adapted to different SIEM platforms with minimal changes to the core detection code, a crucial consideration given GitLab's history of switching SIEMs.

Tines: The SOAR Orchestrator

Tines is central to GitLab's automation efforts, acting as the brain for connecting and automating workflows between GitLab, Slack, Dvo, and other services. Tines uses a no-code/low-code approach, allowing security engineers to build complex automation "stories" using a visual interface. This simplifies the creation and maintenance of intricate workflows that would otherwise require extensive custom scripting.

User Attestation Module (UAM) Workflow

A prime example of Tines in action is the User Attestation Module (UAM), demonstrated with a Personal Access Token (PAT) creation alert:

1. Alert Trigger: When a PAT is created, Dvo (the SIEM) generates an alert and sends it to a specific Tines webhook.
2. Information Extraction: The Tines story ingests the alert, parsing it to extract critical information such as the username and email address of the PAT creator.
3. User Engagement: Tines then orchestrates a Slack bot to reach out directly to the user who created the PAT. The bot presents two options: "This was me" or "I don't recognize it."
4. Automated Triage:

• "This was me": If the user confirms the activity, the alert is marked as benign, and the Tines story can automatically close the alert in Dvo, logging the user's attestation as a comment.
• "I don't recognize it": If the user denies the activity, Tines immediately notifies the SIRT team in a dedicated Slack channel. This flags the alert as potentially malicious, prompting an engineer to investigate. The system also allows for delegation to other team members.
• No Response Escalation: To prevent alerts from languishing, UAM incorporates a race condition mechanism. If a user does not respond within a defined timeframe (e.g., 1 hour), the alert is automatically escalated to the SIRT team. For repeat offenders who consistently fail to respond, the system can even auto-escalate the issue to their manager, ensuring accountability.

1. Contextual Enrichment: To aid engineers, the Slack bot provides immediate context within the alert thread. It queries a Metric DB (a custom database developed by Harjeet Sharma) for historical alerts of a similar type and any associated commands or investigative notes from past incidents. This gives engineers a head start, reducing the time spent gathering initial information.

Incident Management and Metrics

If an alert is deemed a true positive after attestation or investigation, the SIRT engineer can click a "Convert to Incident" button in Slack. This action triggers a Tines story to automatically create a new incident issue in GitLab. GitLab issues serve as the team's single source of truth for incident management. The incident issue is pre-populated with a standardized summary template, pulling available information from the alert, automating much of the initial documentation. These issues support a timeline feature, severity levels (S1, S2, S3), integration with PagerDuty, and custom labels.

When an alert is closed (whether benign or resolved), engineers provide a closing summary and crucial feedback. This feedback includes a "sentiment analysis" or rating for the alert's quality, helping to identify noisy or ineffective detections. Engineers can also report bugs if a detection is missing details, which automatically creates an issue in GitLab for the detection engineering team. All this data is stored in the Metric DB, providing continuous insights into detection efficacy, false positive rates, and overall IR performance. This data is then used to generate weekly reports, assigning a "score" to each detection to guide tuning efforts and resource allocation.

Handovers (Hogi)

For a global 24/7 team, effective handovers are paramount. GitLab developed "Hogi," an automated Slack bot that facilitates asynchronous communication between shifts. At the end of a shift, Hogi prompts the on-call engineer to list alerts or incidents requiring attention. This information is then automatically stored in GitLab by creating an issue, and also posted in a dedicated Slack channel, ensuring the incoming engineer has immediate context and continuity of operations.

Slack for Communication and Organization

Slack is not just for attestation but serves as the primary communication hub. Tines automates the creation of dedicated Slack channels for each incident, ensuring all communication related to a specific event is contained. Furthermore, alert discussions and updates are kept within threads, preventing information overload in main channels.

Demo / Proof of Concept

▶ Watch: Q&A: Strategy for Automated Token Revocation (33:50)

The talk incorporated several live demonstrations and a hands-on workshop for attendees, showcasing the practical application of GitLab's Automated Incident Response framework.

The speakers began by demonstrating the detection as code pipeline. Harjeet Sharma edited a detection rule for Personal Access Token (PAT) creation directly in a GitLab repository. He then created a merge request (MR), which triggered an automated pipeline. This pipeline performed various checks to ensure the rule's quality and adherence to standards. Upon successful completion of these checks, the MR was merged, which in turn triggered another pipeline to automatically deploy the updated rule to their Dvo SIEM. This illustrated the seamless, version-controlled, and automated deployment of detection logic.

Next, the User Attestation Module (UAM) was demonstrated. Harjeet created a PAT in GitLab, simulating a real-world event. Almost immediately, a Tines-powered bot reached out to him via Slack, asking for attestation ("This was me" or "I don't recognize it"). When Harjeet selected "I don't recognize it," the bot promptly notified the SIRT team in a dedicated Slack channel, demonstrating the automated triage and escalation process. The demo also showed how an engineer could then interact with the alert in Slack, convert it into a GitLab incident issue (pre-populated with relevant data), and finally close the alert, providing feedback that fed into their Metric DB.

The workshop portion, titled "Token Heimer," provided attendees with a hands-on experience building their own automation using Tines. Participants were tasked with creating a Slack slash command (e.g., /nuke <token_id>) that would automatically revoke a GitLab PAT. This involved:

1. Setting up a Tines story (workflow) to receive the Slack slash command via a webhook.
2. Parsing the token ID from the command's text body.
3. Making an HTTP DELETE request to the GitLab API's /api/v4/personal_access_tokens/self endpoint, using the provided token ID to revoke it.
4. Adding logic to provide feedback to the user in Slack (e.g., "Token revoked successfully" or "Failed to revoke").

In a subsequent enhancement, the speakers demonstrated how to extend the "Token Heimer" automation to "scope" the token before revocation. This involved making an initial HTTP GET request to the same self endpoint to retrieve information about the token, such as the user_id and associated scopes. Tines then used this user_id to query the /api/v4/users/<user_id> endpoint to get the actual username. This enriched information (e.g., "This token belonged to root and had API scope") was then posted to Slack alongside the revocation confirmation, providing valuable context for the incident responder. This practical exercise underscored the power of Tines in integrating with various APIs and building complex, context-rich automations with minimal coding.

Defensive Implications

▶ Watch: Overview: GitLab's GUARD Comprehensive Automation Platform (64:10)

The GitLab presentation offers profound defensive implications for any organization striving to enhance its security operations:

1. Embrace Comprehensive Automation: Security teams must move beyond automating isolated tasks and adopt a holistic approach. Automating detection deployment, initial triage, user attestation, incident documentation, and handovers frees up human analysts to focus on complex investigations, threat hunting, and strategic improvements, rather than repetitive, mundane work.
2. Implement Detection as Code: Managing detection rules in a version-controlled repository (like GitLab) with enforced review pipelines ensures high-quality, standardized, and auditable detections. This approach promotes collaboration, reduces errors, and allows for rapid, consistent deployment of new detection logic across the SIEM landscape, regardless of its specific vendor.
3. Leverage SOAR Platforms for Orchestration: Tools like Tines are indispensable for integrating disparate security tools (SIEM, communication platforms, incident management systems, threat intelligence feeds) and orchestrating complex workflows. A SOAR platform acts as the central nervous system for security operations, enabling rapid response and adaptive automation.
4. Integrate User Attestation: Implementing a User Attestation Module (UAM) significantly reduces the volume of false positives that reach the SIRT team. By empowering users to validate benign activities, organizations can drastically cut down on alert fatigue, allowing security engineers to prioritize genuine threats. This also fosters a culture of security awareness among employees.
5. Establish a Single Source of Truth: Centralizing all incident-related information in a robust, version-controlled system (like GitLab issues) is critical for global teams and effective handovers. This ensures consistency, provides historical context, and streamlines post-incident reviews, preventing information silos and loss.
6. Drive Decisions with Metrics: Proactive collection and analysis of metrics on alerts and incidents are non-negotiable. Data on alert noise, true/false positives, and detection efficacy provides objective evidence to tune detections, justify investments in tools and headcount, and continuously improve the overall security posture. Without metrics, security teams operate in the dark regarding their effectiveness.
7. Standardize and Streamline Incident Management: Automating incident reporting, initial severity/priority triage, and documentation processes ensures consistent handling of incidents. This standardization is crucial for maintaining operational efficiency and compliance, especially during high-pressure events.
8. Prioritize API-First Development: When evaluating or developing security tools, prioritize those with robust APIs. An API-first approach is fundamental to building an interconnected and automated security ecosystem, as demonstrated by GitLab's integration of Dvo, Slack, and their own platform via Tines.

Key Takeaways

• Comprehensive automation across detection engineering, incident management, and metric generation is crucial for modern Security Incident Response Teams (SIRTs) to combat alert fatigue, scale operations, and ensure consistent processes.
• SOAR platforms like Tines are powerful enablers for integrating disparate security tools (e.g., Dvo SIEM, GitLab, Slack) and orchestrating complex, no-code/low-code security workflows.
• Implementing "detection as code" with a rigorous merge request (MR) and pipeline-driven deployment process ensures high-quality, standardized, and auditable detection rules, with the added benefit of being SIEM-agnostic.
• User Attestation Modules (UAMs) significantly reduce false positives and streamline initial alert triage by engaging users directly to validate suspicious activities, freeing up security engineers for critical investigations.
• Utilizing GitLab issues as a single source of truth for incident management centralizes documentation, communication, and timelines, facilitating seamless handovers and robust post-incident analysis for global teams.
• Metrics-driven security operations are essential; automated collection and analysis of data on alerts and incidents provide crucial insights for tuning detections, justifying resource allocation, and continuously improving the overall security posture.

About the Speaker(s)

Valentine Mairet is a member of the GitLab Security Incident Response Team (SIRT). Her background spans both offensive and blue team security, bringing a well-rounded perspective to incident response. She is passionate about coding, threat hunting, and the outdoors, and a strong advocate for automation as a means to simplify and improve security operations.

Harjeet Sharma is a Staff Security Engineer also within the GitLab Security Incident Response Team (SIRT). His expertise lies in automating security processes and building tools tailored for IR teams. He is credited with developing key components of GitLab's internal automation, including the Metric DB. Harjeet also enjoys traveling in his spare time. Both speakers are integral to GitLab's efforts in building and refining their advanced automated incident response capabilities.