Exploring the AI Automation Boundary for Threat Hunting at Datadog

Exploring the AI Automation Boundary for Threat Hunting at Datadog

Speaker: Arthi Nagarajan, Software Engineer for Internal Threat Detection, Datadog

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 3–4, 2026, San Francisco

YouTube: Watch the full talk

Reading time: ~9 minutes

TL;DR

Datadog's threat hunting team spent six to nine months discovering exactly where AI can and cannot help in a real-world security operations environment. Their Hunting Copilot evolved through multiple architectures — from a naive single agent to a multi-agent orchestrator-expert framework — achieving a 41% reduction in total hunt time during A/B testing. The central lesson: schema field discovery from live data, not documentation, is the key to semantic accuracy, and human experts must remain in the driver's seat.

Introduction

Threat hunting is often described as finding a needle in a haystack, but at a company like Datadog — which processes massive volumes of logs from many different sources with varying schemas — the haystack itself is constantly changing shape. The queries threat hunters generate during a hunt are not disposable; they eventually become production detection rules. Getting them right matters enormously.

Arthi Nagarajan, a software engineer who works closely with Datadog's internal threat intelligence and detection teams, spent the better part of a year asking two deceptively simple questions: what should we automate, and how? The resulting system, the Hunting Copilot, is a production tool that Datadog's hunters use during actual hunts. At [un]prompted 2026, Nagarajan shared the full story: the failures, the pivots, the hard-won lessons about AI's limits, and the concrete performance numbers from live A/B testing.

Her perspective carries particular weight because it's grounded in operational reality. The Hunting Copilot wasn't a research prototype — it was tested during actual "hunt weeks" where hunters were expected to deliver results.

▶ Watch: Introduction and Problem Statement (00:00)

The Data Problem: Why Threat Hunting Resists Automation

Before describing what they built, Nagarajan set up the structural challenges that made threat hunting particularly hard for AI to handle.

Datadog's log environment involves many different sources, variable and often customizable schemas, long historical lookback periods of up to 450 days, and a mix of faceted fields (indexed, easy to filter) and unfaceted fields (raw attributes, not indexed). "It's really tedious to figure out which queries will surface the intended data," she explained.

The deeper problem is the absence of reliable ground truth. Threat hunting success criteria are inherently ambiguous — inputs, outputs, and methods vary dramatically hunt to hunt and hunter to hunter. Without well-curated, representative, consistently labeled historical hunt data, fine-tuning was off the table. "We did not already have a well-curated corpus of past threat hunts with consistent inputs and outputs labeled in a way that can apply to unseen hunts," Nagarajan said. The team needed to deliver results quickly, and fine-tuning carried too much risk and too high a data collection cost.

▶ Watch: The Data Problem (04:00)

Hunting Copilot V1: A Promising Failure

The first iteration used GPT-4.1 as a single agent with a bare-bones system prompt. The prompt linked to Datadog's syntax documentation, provided hypothesis-to-query examples, and essentially said "you're a threat hunting agent — do your thing."

Results arrived quickly: roughly five seconds per response. But quality was poor across every dimension that actually mattered.

V1's failure modes, by the numbers:

• Syntax error rate: 25% (a quarter of all generated Lucene queries were syntactically invalid)
• Aggressive hypothesis production (too many hypotheses, not enough focus)
• Frequently misnamed fields and irrelevant values
• Inability to automatically rule out obvious false positives (like logs associated with known testing accounts)

The four root causes Nagarajan identified: ineffective tool usage (the agent ignored the tools it was given), low syntax accuracy, low semantic accuracy (queries were valid but surfaced the wrong data because field names were hallucinated), and low relevance (the agent wasn't using actual data to drive conclusions).

"We concluded that LLMs are bad at threat hunting," she said, "and we had to go back to the basics and understand what their capabilities actually were." Those capabilities — synthesis, pattern matching, translation — are real but narrow. The gap was specifically in understanding the actual schema of the live data environment.

▶ Watch: Hunting Copilot V1 Architecture and Failures (02:00)

The Reframe: Log Schema Exploration as Source of Truth

The critical insight that unlocked V2 came from watching actual threat hunters work. Their process turned out to be repeatable: start with a broad hypothesis and initial queries, dive into raw logs and inspect sample values, tweak the query, and repeat. Nagarajan called this "log schema exploration" — treating the live data itself, not documentation, as the authoritative source of schema truth.

This reframe addressed the core problem directly. Instead of asking the agent to guess field names based on documentation, the team shifted the system toward dynamically learning schema from real queries and real results. "Instead of relying on documentation as a source of truth, the schema should be derived directly from the data."

This insight drove four approaches to schema learning that the team continues to balance:

1. External documentation + AI — educated guesses, not representative of actual data in practice
2. Log clustering — cuts down log volume but reduces relevancy
3. Preprocessing with ingestion pipelines — static context, subject to frequent change
4. Live sampling — dynamically learns schema from actual data but is computationally exhaustive

No single approach dominates. The production system coordinates across all four.

▶ Watch: The Schema Exploration Pivot (06:00)

Hunting Copilot V2: Multi-Agent with MCP Integration

The improved architecture introduced several significant changes: reasoning models (GPT-5.1 and GPT-4.1), an orchestrator-expert multi-agent framework, and critically, the Datadog MCP server — which allowed the agent to actually execute the queries it generated and validate them against real logs.

The Datadog MCP server was the single most impactful addition. When a query failed, the MCP returned concise, structured corrections rather than raw error output — removing the burden of self-correction from the agent and providing specific, actionable feedback. This drove a 17% improvement in syntax accuracy, bringing the overall syntax accuracy to 92%.

The orchestrator-expert structure divided labor intelligently: the orchestrator built context across the full hunt, while sub-agents had their context reset for each specific task — keeping per-agent context manageable. Stop conditions were both quantitative (cap iterations at five per query) and qualitative (if you've found logs and can reach a conclusion, stop).

But V2 came with a painful tradeoff: response time exploded from five seconds to six minutes average, with some edge cases taking thirty minutes to an hour. "Our users were not happy," Nagarajan admitted. Semantic accuracy also remained limited by persistent hallucination, because query executions frequently timed out or returned truncated results — making live log sampling too computationally expensive for the agent to handle alone.

▶ Watch: Hunting Copilot V2 Architecture and Results (08:00)

Hard Lessons from Multi-Agent Design

The lessons Nagarajan extracted from V2's performance have broad applicability beyond threat hunting:

Reasoning models are too slow for most steps. Using reasoning models across the board added latency without proportionate accuracy gains for straightforward tasks. Reserve reasoning for genuinely complex steps.

Limit agent chain depth to three layers. Chains longer than three steps were prone to getting stuck in loops, particularly during iterative query refinement. The team enforced a three-layer-or-less architectural constraint.

Tools must be concise and informative. Tools designed to return short, specific suggestions — rather than raw data dumps — dramatically reduced the agent's burden. MCP servers that expose many tools should have a curated allowed-tool list specified in the system prompt, with explicit instructions on which tool to use at which step.

Context is everything. The consistent lesson across every V2 failure was context management. "If you manage the context well, you'll see results improve," she said. Semantic accuracy problems traced back to context overload causing truncation and hallucination.

▶ Watch: Multi-Agent Trade-offs and Lessons (10:00)

Evaluation: When Benchmarks Don't Map to Reality

One of the talk's most valuable sections addressed the deep dysfunction in standard AI evaluation approaches when applied to threat hunting. The "data problem" that made fine-tuning impractical applies equally to evaluation.

Traditional evaluation systems — fixed inputs mapped to expected outputs — produced misleading results. The team generated synthetic evaluation data, ran evaluations showing high accuracy, and then watched actual hunters reject the outputs. "Passing evaluations did not always correlate with trust," Nagarajan said. "A single number to capture the performance can be quite misleading."

Their realistic evaluation strategy evolved to include:

• A/B testing during hunt weeks — humans with the tool versus humans without, comparing actual hunt results
• Tight user feedback loops — frequent communication about behavioral inadequacies
• Atomic task evaluation — measuring response speed (objectively) and syntax accuracy (via the Datadog syntax validation checker) separately from semantic accuracy, which required manual verification rather than LLM-based judging
• Usage as a proxy for success — if hunters chose to use the tool, it was working

For semantic accuracy specifically, they explicitly avoided using an LLM judge because the same hallucination problems that hurt the agent would infect the judge. "That would be circular," she explained.

▶ Watch: Evaluation Strategy (12:00)

Production Results: An AWS Privilege Escalation Hunt

Nagarajan walked through a concrete real-world hunt to show the tool in action. One hunter was investigating an IAM escalation path using the iam:CreateLoginProfile event.

The Hunting Copilot generated relevant Lucene queries, automatically broadened scope from one day to seven days to thirty days as it refined its approach, identified accounts associated with known test users, interpreted results correctly, and provided a verdict of "likely benign" along with next steps — specifically naming which users the hunter should contact to verify authorization. When the hunter asked a follow-up question about subsequent console logins for created users, the agent generated a new query and determined there was no evidence of that activity, covering all relevant users it had identified in the prior step.

A/B Testing Results (AWS Privilege Escalation Hunt Week):

• Average query iteration time cut from 10 minutes to 5 minutes
• Average hunt time saved per hypothesis: 25 minutes out of a 60-minute average
• 41% reduction in average total hunt time
• Response times remained under six minutes throughout the week

The hunt week also produced two new cloud detection rules and uncovered zero actual incidents — exactly what good threat hunting often looks like.

▶ Watch: Production Hunt Demo and A/B Testing Results (14:00)

The Current Automation Boundary

Based on all this work, Nagarajan defined Datadog's current automation boundary as threefold: query generation, data analysis, and pivot across sources. But critically, she was clear about what AI should not do: drive the hunt strategy. Threat hunters should build playbooks and define hypotheses. The AI handles the mechanical complexity of diverse telemetry and platform-specific syntax.

"Threat hunters lead the hunts," she said. "Provide an accurate understanding of the data's schema. Field discovery increases trust."

▶ Watch: The Automation Boundary (18:00)

Notable Quotes

"We concluded that LLMs are bad at threat hunting. We had to go back to the basics and understand what their capabilities actually were." — Arthi Nagarajan (02:00)

"Instead of relying on documentation as a source of truth, the schema should be derived directly from the data." — Arthi Nagarajan (06:00)

"Passing evaluations did not always correlate with trust. A single number to capture the performance can be quite misleading." — Arthi Nagarajan (12:00)

"If you manage the context well, you'll see results improve." — Arthi Nagarajan (08:00)

Key Takeaways

• Schema field discovery is the hardest part of AI threat hunting. Syntax accuracy is achievable (92% with MCP-powered validation), but semantic accuracy — knowing which actual field names and values exist in your specific environment — requires live data sampling, not documentation.
• Multi-agent frameworks improve quality but destroy latency. V2 went from 5-second to 6-minute average response times. Architect for the specific steps that truly need reasoning; don't apply it everywhere.
• Humans must drive the strategy. The AI automation boundary ends at playbook creation and hypothesis definition. The tool works best when a hunter is in the driver's seat with a concrete playbook.
• Standard benchmarks lie. Build evaluation around actual usage patterns — A/B testing during real hunts, tight user feedback loops, and treating usage as a success signal.
• Limit agent chain depth. Chains longer than three steps become unstable. Three layers or fewer is a practical engineering constraint, not just a performance preference.

Slides Reference

Slides are available at http://bit.ly/4kZQYB4. The slides PDF (from 2026-04-04-D2-S1-15-10-Exploring-the-AI-Automation-Boundary-f.pdf) covers: problem definition, the V1 architecture using GPT-4.1, V1 failure modes with quantitative results, the log schema exploration reframe, V2 orchestrator-expert architecture with Datadog MCP, V2 performance metrics (92% syntax accuracy, 6-minute average response time), four semantic accuracy approaches, the realistic evaluation strategy, the IAM escalation hunt case study, A/B testing results (41% hunt time reduction), and the final automation boundary definition.