How to Train Your Detection Dragon

How to Train Your Detection Dragon

Speaker: Geet Pradhan

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch on YouTube

Reading time: ~7 minutes

TL;DR

Building a detection and response pipeline from scratch is less a technology problem than a design and communication problem. Geet Pradhan, a security engineer speaking from hard-won experience, walks through a four-phase framework — sources, alerting, notification, and response — and explains the organizational and cultural patterns that separate a pipeline that scales from one that collapses under its own alert volume.

Introduction

The "Here Be Dragons" theme of BSidesSF 2025 was tailor-made for Geet Pradhan's talk, which frames detection response as a multi-headed creature to be trained rather than slain. Drawing on lessons learned building out a detection pipeline at a company with an already-functional tech stack, Pradhan delivered a talk aimed at the security engineer who has just been handed the SecOps mandate and is staring at a blank whiteboard.

Detection response is, as Pradhan put it with understatement, "easy — said no one ever." The challenge is not any single piece of the pipeline: it is the interaction between sources, rules, notifications, and the humans who respond to alerts at 2am. Most pipeline failures happen not because a detection missed something, but because the response side was not designed with enough care to handle what the detection side surfaced. Pradhan's talk addresses both.

Phase 1: Where Do You Start? Prioritize Your Log Sources

▶ Watch: Log source prioritization (04:00)

The first instinct when building a detection pipeline is to ingest everything. Pradhan's experience is that this is both impractical and counterproductive — not every log source deserves equal priority, and ingesting low-value sources early crowds out the engineering time needed to make high-value ones actionable.

His prioritization framework proceeds in four tiers:

1. Compliance-scoped systems first. If a system falls under a regulatory framework (SOC 2, ISO 27001, PCI-DSS), it should be ingested first. Regulatory obligation creates natural leverage with stakeholders and ensures the most scrutinized systems are monitored.
2. Production and business-critical systems outside compliance scope. Some systems are essential to operations but fall below the compliance threshold. A critical internal production app that is not formally in scope still warrants early ingestion.
3. Cross-functional tier-1 systems. Once the compliance and business-critical sources are in, consult sister teams: "We have ten or twenty sources ingested — what are your P1 concerns?" This surfaces systems the security team might not be aware of and builds goodwill.
4. Everything else — when time and resources allow.

Pradhan's point is that reaching tier three already represents substantial coverage for most organizations at this stage. The fourth tier is an aspiration, not a requirement.

Phase 2: Setting Alerts — The Goldilocks Principle

▶ Watch: The Goldilocks principle for alerting (06:00)

Too few alerts and threats slip through. Too many and the team becomes desensitized — alert fatigue makes everything invisible. The Goldilocks principle Pradhan describes is about finding the alert volume the team can actually act on.

The practical approach: talk to system owners before writing detection rules. Pradhan described discovering that several engineering teams were routinely bypassing branch protection rules in GitHub — not as a security incident, but as their standard development workflow. Without that conversation, the detection would have generated constant noise and eventually been suppressed. With it, the team could write a targeted rule that captured genuinely anomalous bypass events while excluding the expected pattern.

For the detection logic itself, Pradhan is a strong advocate for Sigma rules — an open, vendor-agnostic detection format that is friendly to developers and can be adapted across SIEM platforms. He referenced a talk by Alex Sinnott of Wise as a practical guide to implementing Sigma in production. Pradhan also pointed to the MITRE ATT&CK framework and free detection libraries as starting points for threat coverage: "You don't have to write everything from scratch — get an understanding of what detection visibility you need, then tune from there."

Phase 3: Notification Schedules — The Right Alert in the Right Channel

▶ Watch: Notification and severity mapping (12:00)

A detection that fires but lands in the wrong channel — or one nobody monitors — is functionally the same as no detection at all. Pradhan's notification strategy is built around a severity mapping table:

• Critical: PagerDuty page. Drop everything. Response required immediately.
• High: Slack, with explicit tagging of the responsible individual. Needs attention promptly but not a middle-of-the-night wake-up.
• Medium/Low: Slack channel posts, no tagging. Review during morning triage.

The principle behind this structure is using channels the team already lives in. Pradhan argued against introducing new alerting platforms or requiring engineers to log into a SIEM to respond to notifications. "Use Slack, use email — in today's world we're never just at our machine. You wanna see an alert quickly on your phone."

The Sigma rule format also supports embedding response playbook links directly in the rule definition, so when an alert fires, the responding engineer sees not just the event but the specific steps to triage it. "Everything you need to respond is in the rule itself."

Phase 4: Scaling the Pipeline — Enrichment, Exceptions, and Automation

▶ Watch: Log enrichment and pipeline scaling (18:01)

After the initial pipeline is live, Pradhan described a predictable failure mode: the response side breaks under alert volume. He was frank about his own experience: "I had a thousand alerts a day and I wanted to sleep." The solution is a set of techniques for contextualizing and filtering alerts before they reach a human.

Log enrichment means adding metadata to raw events so that alerts carry enough context to triage without additional investigation. In Pradhan's environment, a GitHub alert that triggers on a repository event is enriched with the repository's business criticality score — if the repository is not production-critical, the alert is de-escalated automatically rather than requiring an engineer to look it up manually.

Exceptions handle known-good patterns that would otherwise fire continuously. The important nuance here is that exceptions should reduce severity, not silence alerts entirely. Pradhan's example: an administrator who regularly performs a privileged action (say, assigning a Google Workspace role) from a known-good location could have their alerts reduced from critical to high. The alert still fires and is still visible; it just does not wake someone up at 3am.

Automated user context gathering is another high-leverage tactic. When an alert fires, the automation tags the relevant user in Slack and asks them to confirm whether the action was intentional. This technique works across time zones — if Pradhan is in London and the user is in San Francisco, the user can respond during their own working hours, so by the time Pradhan wakes up, the context is already there. "I don't have to wait another eight hours for Teddy to be awake."

▶ Watch: Automation and cross-timezone triage (24:01)

Sustaining the Pipeline: Cross-Functional Trust

The final section of the talk addressed the hardest part of security operations: getting the rest of the company to participate. Detection response is a team sport, and most of the critical signals come from actions taken by engineers in other organizations.

Pradhan's approach to cross-functional security is built on two principles. First, security champions — people embedded in engineering, infrastructure, or product teams who can translate security requirements into their team's language and advocate from within. Second, a "yes, and" posture — rather than telling teams they cannot do something, tell them how to do it safely. "Security is thought of as the police. Hey, you're not supposed to do this. Instead of that, say, 'Okay, you wanna do that? Add guardrails one, two, three, four.'"

The practical implication for detection is that process complexity should be invisible to the people triggering the alerts. If the response requires an engineer outside the security team to log into a new tool or fill out a custom form, the response rate will be low. If it just means responding to a Slack message, it works.

For tools, Pradhan recommended the open-source stack: Wazuh as an open-source SIEM, Tracecat as an open-source SOAR for automation, and the Sigma Rules repository for detection coverage. MITRE ATT&CK rounds out the toolkit for understanding attacker behavior and structuring response playbooks.

Notable Quotes

"Detection response is easy — said no one ever." — Geet Pradhan (02:00)

"I would not have been able to set up my detection pipeline if I did not work with the system owners." — Geet Pradhan (08:00)

"Security is thought of as the police. Instead, be a collaborator — the first thing you need from them is you need them to listen to you." — Geet Pradhan (28:02)

Key Takeaways

• Prioritize log ingestion by tier — compliance-scoped systems first, then business-critical, then cross-functional, then everything else. Reaching tier three already represents strong coverage.
• Talk to system owners before writing detection rules. What looks like malicious behavior is often an undocumented but legitimate workflow — and knowing the difference prevents both false positives and organizational friction.
• Sigma rules provide a vendor-agnostic, developer-friendly format for detection logic that can embed response playbooks and be deployed through a CI/CD pipeline.
• Severity mapping drives notification design. Critical events page; high events tag; medium and low go to monitored channels. Use the tools your team already lives in.
• Scale with enrichment and exceptions — add business context to alerts, reduce severity for known-good patterns, and use automation to gather user context before a human is needed.
• Build cross-functional trust through security champions and a "yes, and" posture. Security teams that are seen as collaborators get better signal and faster responses than those seen as blockers.