Agents That Watch: Event-Driven Detection and Response at Slack

Dominic Marks (Slack)

[un]prompted.au 2026 · Day 1 · Main Hall

Overview

In an era of increasingly sophisticated cyber threats, traditional security detection methods often struggle to keep pace with dynamic attack techniques and the sheer volume of data. Dominic Marks from Slack Security presented an insightful talk at [un]prompted.au, "Agents That Watch: Event-Driven Detection and Response at Slack," detailing Slack's innovative approach to protecting customer data through a multi-agent, AI-powered system. This system automates the characterization, investigation, and reporting of security incidents, designed to provide a proactive and adaptive defense mechanism.

Watch on YouTube

Visual summary for Agents That Watch: Event-Driven Detection and Response at Slack by Dominic Marks
Visual summary for Agents That Watch: Event-Driven Detection and Response at Slack by Dominic Marks

Key moments

  1. 0:00 Introduction: Slack's agent-based detection and response
  2. 6:00 Watchdog agents: Clustering signals and LLM-driven SQL
  3. 10:00 Investigation teams: Parallel workstreams and cross-verification
  4. 15:30 Red team engagement: Analyzing watchdog detection failures
  5. 18:00 Nave: Session store for reproducible agent experiments
  6. 24:30 Future improvements: Refining watchdog agent architecture
  7. 28:00 Q&A: Benefits over traditional SIEM alerts

Agents That Watch: Event-Driven Detection and Response at Slack

Speakers: Dominic Marks, Slack

Conference: [un]prompted.au

YouTube: https://www.youtube.com/watch?v=vQKnps-UdR4

Overview

In an era of increasingly sophisticated cyber threats, traditional security detection methods often struggle to keep pace with dynamic attack techniques and the sheer volume of data. Dominic Marks from Slack Security presented an insightful talk at [un]prompted.au, "Agents That Watch: Event-Driven Detection and Response at Slack," detailing Slack's innovative approach to protecting customer data through a multi-agent, AI-powered system. This system automates the characterization, investigation, and reporting of security incidents, designed to provide a proactive and adaptive defense mechanism.

The core of Slack's system revolves around three distinct populations of AI agents: Observers, Watchdogs, and Investigation agents. Observers are responsible for ingesting and classifying events from various infrastructure sources, building a narrow, rich view of specific system components. Watchdogs then consume these signals, looking for broader patterns indicative of threats across the entire infrastructure. Should a Watchdog identify a cluster of concern, it dispatches the matter to Investigation agents, which conduct deep, verified analyses to determine the security implications and escalate actionable reports to human analysts. This talk not only outlines the architecture of this sophisticated system but also critically examines its performance against a real-world red team engagement, highlighting both successes and areas for improvement, underpinned by a unique reproducible experimentation framework.

The significance of this talk lies in its demonstration of how large language models (LLMs) and autonomous agents can be leveraged to build a more resilient and responsive security posture. By delegating complex threat anticipation and analysis to AI, Slack aims to overcome the limitations of manually crafted detection rules that struggle to account for unforeseen attack vectors. The system's ability to learn and adapt from past investigations, coupled with its robust verification mechanisms, represents a significant step forward in automated security operations, offering valuable insights for organizations grappling with similar challenges in scaling their defensive capabilities.

Background

▶ Watch: Introduction: Slack's agent-based detection and response (0:00)

The landscape of cybersecurity threats is constantly evolving, with adversaries becoming more sophisticated, often employing automated or "cyber-capable models" themselves. This presents a fundamental challenge for traditional security operations, which typically rely on Security Information and Event Management (SIEM) systems and static detection rules. As Dominic Marks highlights, the inherent limitation of these traditional methods is their reliance on anticipating "what's coming." Security analysts must imagine and codify every potential attack vector, a task that becomes increasingly impossible as attack surfaces expand and threat actors innovate. This reactive posture often leaves organizations vulnerable to novel or rapidly changing threats that fall outside predefined detection logic.

The problem is exacerbated by the sheer volume of telemetry data generated by modern cloud infrastructure and services. Sifting through millions of logs to identify subtle indicators of compromise (IOCs) is a monumental, often human-intensive, task. This leads to alert fatigue, missed signals, and delayed response times. Slack's motivation to develop an agent-driven system stems from this recognition: a need to move beyond static, rule-based detection to a more dynamic, intelligent, and autonomous approach. By delegating parts of the detection and response workflow to LLMs and autonomous agents, Slack aims to create a system that can not only process vast quantities of data but also "learn" and "reason" about potential threats, reducing the burden on human analysts and improving overall responsiveness. This paradigm shift allows the security team to focus on higher-level strategic defense and incident response, rather than constantly updating detection rules for known attack patterns or sifting through noise.

Key Findings

▶ Watch: Investigation teams: Parallel workstreams and cross-verification (10:00)

The Slack Security team's agent-driven system represents a significant advancement in automated detection and response, demonstrating both the immense potential and the practical challenges of deploying AI in critical security functions. A pivotal "key finding" from the talk emerged from a real-world red team engagement conducted in June. While the system's Observers and Investigation agents performed effectively, the Watchdog component revealed critical operational and logical failures that prevented end-to-end detection.

Specifically, two distinct issues were identified during the red team exercise:

  1. Bookkeeping Error: A Watchdog failed to properly monitor and respond to signals indicating a "burst of repository cloning activity" produced by an Observer. Despite the Observer correctly identifying the anomalous behavior, the Watchdog's internal logic overlooked these signals.
  2. Scheduling Automation Failure: Another Observer correctly identified "workflow execution logs" being deleted—a common anti-forensics technique. However, due to a failure in the Watchdog's scheduling automation, the Watchdog was not running during a critical period, leading to a complete miss of these crucial signals.

Despite these Watchdog failures, an analyst independently identified the malicious behavior and initiated an incident response. Crucially, the Investigation team was later engaged and successfully produced a comprehensive report, validating the effectiveness of the initial and final stages of the agent pipeline.

Post-engagement analysis, facilitated by Slack's unique Nave session store, allowed for a critical counterfactual experiment. By re-running the system with the production snapshot from June but without the scheduling failure (P0 experiment), it was found that in 7 out of 10 runs, the system would have successfully escalated to the investigation team. This validated the core potential of the end-to-end system, confirming that the primary failure was operational (scheduling) and a logical bug (bookkeeping) rather than a fundamental flaw in the agent design. Further experiments with updated code (C0) yielded similar results, while minor prompt adjustments showed a slight improvement (8/10), though the sample size was too small for definitive conclusions.

A significant insight from these experiments was the Watchdog's biased signal integration: it heavily favored alerts related to workflow log deletions over repository cloning activities. This indicated that while the Watchdog was capable of identifying individual signals, its ability to properly integrate and weigh different types of evidence across its broad remit needed refinement. The ability to conduct these reproducible experiments using Nave proved invaluable for diagnosing failures, validating fixes, and iteratively improving the agent's performance and decision-making logic. This demonstrates that while AI-driven systems offer immense potential, continuous calibration and rigorous testing are essential for their reliability in high-stakes security contexts.

Technical Deep Dive

▶ Watch: Red team engagement: Analyzing watchdog detection failures (15:30)

Slack's event-driven detection and response system is built upon a sophisticated architecture comprising three interconnected agent populations and a foundational session store called Nave. Each agent type is designed with a specific scope and purpose, contributing to a layered defense mechanism.

Observers: The Eyes and Ears

Observers are the first line of defense, acting as event-triggered, persistent agents. There are approximately 5,000 such agents in Slack's infrastructure, each dedicated to monitoring a specific function or resource. For example, an Observer might be assigned to a particular repository like "Acme Web" to ingest audit events from the source control system, or to an S3 bucket to monitor policy changes, or an IAM policy. When relevant events occur, they are routed to the dedicated Observer.

A key feature of Observers is their structured memory, which is persistent across invocations. This memory allows an Observer to build a rich, narrow view of its assigned resource, remembering past states and activities. This approach differs from traditional event processing by enriching data not just with external context but also with the agent's internal memory and a pre-trained context model. The output of Observers are signals, which classify observed events based on a common schema. This schema includes fields like principal (the actor), subject (the modified entity), annotations (e.g., hostnames), and critically, security belief expressed using MITRE ATT&CK techniques. This allows the agent to articulate what it believes is happening in a given event, providing a standardized language for downstream analysis. Free text is also included for additional context.

Watchdogs: The Pattern Seekers

Watchdogs are autonomous agents responsible for a specific remit, which can be defined by a MITRE ATT&CK tactic, a particular service, or any potential risk or threat model. Unlike Observers, Watchdogs operate on a scheduled scan, consuming signals produced by all Observers and other security agents. Their primary job is to look for patterns, cluster related signals, and make judgments about these clusters.

Similar to Observers, Watchdogs possess structured memory, which they use to learn about the threats they encounter over time. LLMs play a crucial role here, as they are "extremely good at writing SQL," enabling Watchdogs to dynamically generate queries to navigate and aggregate the vast pool of signal data stored in a SQL database. Based on their analysis, Watchdogs produce two types of signals:

  • Dispatch: If a cluster of concern is identified, it is dispatched to an Investigation team for deeper analysis.
  • Defer: If the cluster is not deemed immediately concerning, it is deferred, potentially to be revisited later.

Watchdogs can also dynamically adjust their execution rate. If they detect concerning activity in their environment, they can choose to run more frequently, or back off to save costs if things are quiet. This adaptive scheduling enhances both responsiveness and resource efficiency.

Investigation Teams: The Deep Divers

Investigation teams are activated either by an analyst or by a Watchdog dispatch. Each investigation is assigned a case containing source data, references, and specific objectives or questions. To manage costs, investigations operate within a predefined budget, expressed in terms of tool calls.

The process within an investigation team is highly parallelized and verifies:

  1. Lead Agent: Receives the case and partitions the investigation into independent, parallel workstreams. It then writes narrowly scoped prompts and launches Coordinator agents.
  2. Coordinator Agents: Multiple Coordinators typically run in parallel. They delegate specific tasks to Specialist agents.
  3. Specialist Agents: Each Specialist is assigned a single data source (often an MCP server), a budget, and a security-relevant question. They perform their analysis and produce provisional findings.
  4. Cross-Verification: A critical aspect of investigations is the emphasis on verification. Coordinators are encouraged to use multiple Specialists to cross-verify findings. If a Specialist returns a negative finding or an unexpected result, the Coordinator might launch a new Specialist with a different methodology or data source. Similarly, the Lead agent can launch additional workstreams for cross-verification.
  5. Interactive Element: Analysts can interact with the Lead agent via Slack, providing guidance or pausing the investigation as needed.
  6. Budget Management: The Lead agent dynamically manages the budget, terminating ongoing workstreams and reclaiming budget if new verified findings reshape the investigation's direction.

Ultimately, the Lead agent receives verified findings from the Coordinators and synthesizes them into a detailed report, which is then escalated to human analysts in Slack for incident response. This deep analysis, costing "a lot of tokens," is crucial because the outcomes of these investigations are used to calibrate the Watchdogs, making them more sensitive to confirmed incidents and less prone to false positives (deferring benign or authorized behavior).

Nave: The Foundation for Reproducibility

Underpinning the entire agent system is Nave, Slack's proprietary session store. Nave is a centralized, persistent database and API that acts as a work queue for all agents. When a new agent is defined, an entry is created in Nave. Harness applications poll Nave's API for runnable sessions, claim them, progress them through inference, and then release them back.

Nave automatically streams an event stream during every agent execution, capturing every detail: inference requests, tool calls, and application-layer signals. This provides a permanent, comprehensive audit log. Furthermore, Nave stores snapshots, which are resumable captures of the harness application's state, including the agent's structured memory (often a JSON object). This capability allows for the complete reconstruction and re-execution of any agent from any point in time. This is invaluable for debugging, post-mortem analysis, and performing counterfactual experiments, as demonstrated during the red team engagement analysis. Nave effectively functions as an MCP server itself, providing a powerful interface for other agents to query its data, creating an agent-assisted workflow for system development and analysis.

Demo / Proof of Concept

▶ Watch: Future improvements: Refining watchdog agent architecture (24:30)

The most compelling "demo" of Slack's agent system came not from a staged presentation, but from its real-world performance during a red team engagement in June. This practical test case exposed both the strengths and weaknesses of the integrated agent pipeline.

During a 48-hour period of the engagement, Observers successfully produced critical signals. For instance, one Observer detected a "burst of repository cloning activity," a potential indicator of data exfiltration. Another Observer identified "workflow execution logs" being deleted, a common anti-forensics technique. These findings confirmed the Observers' ability to accurately identify concerning behaviors at the infrastructure level.

However, the Watchdog component, intended to bridge the gap between Observer signals and deeper investigations, failed in two significant ways:

  1. Bookkeeping Error: The Watchdog, despite observing the repository cloning signals, suffered from a "bookkeeping error in its logic" and "skipped over them," failing to properly respond.
  2. Scheduling Failure: For the workflow log deletion signals, the Watchdog's "scheduling automation... failed," resulting in an "extended period of time where the watchdog wasn't running." Consequently, these critical anti-forensics findings were not processed.

These two failures meant the system did not achieve its goal of end-to-end detection via the automated pipeline. Fortunately, a human analyst independently identified the red team's behavior and initiated an incident response. This human intervention eventually led to the engagement of the Investigation team, which successfully "produced a good report," further validating the effectiveness of the deep analysis agents.

The true "proof of concept" of the system's robustness and the value of its design came in the post-mortem analysis, facilitated by the Nave session store. Dominic Marks described a meticulous process of performing counterfactual experiments to understand what would have happened if the operational and logical failures hadn't occurred.

  1. Experimental Setup: Using Nave, a snapshot of the Watchdog agent from production in June was loaded into a dedicated, isolated Nave instance. This allowed the agent to be re-executed against the production signal dataset (Nave acts as an MCP server for signals) while keeping its outputs isolated.
  2. P0 Experiment (Production 0): This test simulated the Watchdog's behavior with the June code and configuration, but without the scheduling automation failure. Across 10 runs, the Watchdog successfully escalated to the investigation team in 7 out of 10 cases. This demonstrated that, given the opportunity to run, the Watchdog could have detected the red team activity, confirming the primary issue was an operational fault.
  3. C0 Experiment (Current 0): Running the same test with the latest September 1st code and configuration showed no significant improvement or degradation (also 7/10 escalations), indicating the core logic remained similar.
  4. Prompt Adjustments: Two variations of the Watchdog's prompt were tested, achieving 8 out of 10 escalations. While a small sample size, this hinted at the potential for prompt engineering to refine agent behavior.

The analysis also revealed a bias in the Watchdog's current form: it was "very heavily biased towards these workflow log deletions rather than the repository clones." This highlighted a challenge in how the Watchdog integrates disparate signals, indicating a need for better balancing of evidence types.

This entire analysis process was itself an agent-assisted workflow. Marks used a "coding agent" attached to Nave to query the session data, another agent to build the test protocol, an "orchestration agent" to spin up and manage the isolated Nave instances for the experiments, and a final agent to collect and assemble the results. This demonstrated the power of the Nave platform not just for detection, but for continuous development, debugging, and validation of the agent system itself.

Defensive Implications

▶ Watch: Q&A: Benefits over traditional SIEM alerts (28:00)

Slack's approach to event-driven detection and response offers several critical defensive implications for organizations looking to enhance their security posture:

  • Embrace AI for Dynamic Threat Detection: Moving beyond static, rule-based SIEM alerts is crucial. The system demonstrates that LLMs and autonomous agents can be leveraged to anticipate and respond to threats that are difficult to pre-define, especially against adversaries employing sophisticated, potentially AI-driven, attack techniques. This delegation of threat anticipation frees human analysts for more strategic work.
  • Implement a Layered, Specialized Agent Architecture: The three-tiered agent model (Observers, Watchdogs, Investigation Teams) provides a robust, complementary defense. Observers offer narrow, deep visibility into specific infrastructure components; Watchdogs provide broad, pattern-based threat modeling; and Investigation teams conduct deep, verified analysis. This specialization ensures comprehensive coverage and efficient resource allocation.
  • Prioritize Reproducibility and Calibration: The Nave session store highlights the critical importance of building security systems with inherent reproducibility. The ability to completely reconstruct and re-execute agent behavior from any point in time is invaluable for debugging, post-mortem analysis (as demonstrated by the red team engagement), and validating improvements. Furthermore, using the outcomes of deep investigations to calibrate Watchdogs (making them more sensitive to confirmed incidents and less prone to false positives) is essential for maintaining trust and effectiveness in AI-driven systems.
  • Leverage Structured Memory and Context: Agents with persistent structured memory can build rich, evolving views of their environment and learn from past experiences. This allows for more intelligent enrichment of events and pattern recognition, moving beyond stateless processing. The use of MITRE ATT&CK for expressing security belief provides a standardized, actionable language for agents to communicate findings.
  • Dynamic Resource Management: The Watchdogs' ability to dynamically adjust their execution rate based on perceived threat levels offers both cost savings and improved responsiveness. This intelligent resource allocation is vital for managing the operational overhead of a large-scale agent system.
  • Integrate Verification Loops: The emphasis on cross-verification within the Investigation teams, involving multiple agents and methodologies, is a crucial design pattern for ensuring the accuracy and reliability of AI-generated findings. This minimizes the risk of agents making mistakes and escalating false positives.
  • Agent-Assisted Development and Operations: The demonstration of using agents to assist in the analysis, testing, and orchestration of the agent system itself (e.g., coding agents interacting with Nave) showcases a powerful meta-capability. This paradigm can significantly accelerate the development, deployment, and continuous improvement of complex AI-driven security solutions.

Key Takeaways

  • Multi-Agent Layered Defense: Slack employs a sophisticated three-tier agent architecture—Observers (narrow focus, persistent memory), Watchdogs (broad threat modeling, pattern recognition), and Investigation Teams (deep, verified analysis)—to create a comprehensive and adaptive security posture.
  • Reproducibility is Paramount: The Nave session store is foundational, enabling complete reconstruction and re-execution of agent behavior. This capability is critical for debugging, post-mortem analysis (as demonstrated by the red team engagement), and continuous system calibration and improvement.
  • LLMs for Enhanced Automation: Large Language Models are effectively leveraged for tasks like dynamic SQL query generation for data navigation and aggregation, significantly enhancing the Watchdogs' ability to identify patterns across vast datasets.
  • Verification and Calibration are Crucial: The system emphasizes extensive cross-verification within investigation teams to ensure the accuracy of findings. The outcomes of these investigations are then used to calibrate Watchdogs, helping them learn from past incidents and reduce false positives.
  • Adaptive and Intelligent Agents: Agents utilize persistent structured memory to build context and learn. Watchdogs can also dynamically adjust their execution frequency based on perceived threat levels, optimizing for both responsiveness and cost efficiency.
  • Delegating Threat Anticipation: The core benefit lies in delegating the complex task of anticipating and detecting novel threats to AI, moving beyond the limitations of static detection rules and enabling human analysts to focus on higher-level incident response and strategic defense.

About the Speaker(s)

Dominic Marks works for Slack Security, where he is part of the team responsible for protecting Slack's data and customer data. His work involves data ingestion and response, specifically focusing on building and maintaining the agent-driven system discussed in this talk. The development of this sophisticated solution has been an ongoing effort for approximately a year and a half, involving a team of nine people. Notably, Dominic mentioned that "Claude [an LLM] wrote most of the code," albeit "with a lot of tuition," highlighting the significant role of AI in the development process itself.

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

Everyone is selling AI SOC analysts, but Slack actually built one and had the stones to admit it totally missed a red team engagement in production. They save the day by showcasing Nave, their deterministic replay engine, proving exactly why it failed and how you actually debug autonomous agents. This is the antidote to LLM marketing bullshit.

Heather Calloway (CISO) — SOLID

Slack’s security engineering team demonstrates a three-tier LLM agent architecture for automated detection, complete with the scars of a real-world red team failure. It is a mandatory watch for any security leader looking to scale their SOC with AI without buying a vendor's black-box promise. The most valuable takeaway isn't the AI itself, but the deterministic state-replay architecture they built to audit it.

→ Top-rated talks at [un]prompted.au 2026

All talks from [un]prompted.au 2026