Your Intrusion Detection Still Sucks (And What to Do About It)

Your Intrusion Detection Still Sucks (And What to Do About It)

Speaker: Jason Craig

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch on YouTube

Reading time: ~8 minutes

TL;DR

Detection and response teams are drowning in low-fidelity alerts, letting attackers dwell for ten to fifteen days on average before detection. Jason Craig, Director of Detection and Response at Remitly, argues that the fix is not more alerts — it is context, decoration, cross-correlation, and enrichment applied to raw events before anything reaches an on-call engineer. The result: fewer meaningless pages, faster triage, and detections that hold up even when adversaries try to hide.

Introduction

In 2023, Jason Craig gave a talk at BSidesSF titled "Lapsus$ is Winning," arguing that the industry's detection posture was losing ground against a new generation of adversaries — many of them unsophisticated teenagers — who kept finding ways in. Two years later, Craig returned with a follow-up that acknowledges incremental progress on dwell times while insisting the underlying architecture problem remains largely unsolved.

The data Craig cited from the just-released 2025 Verizon DBIR and Mandiant M-Trends reports is sobering: median attacker dwell time for a confirmed breach is ten to fifteen days; internal detection (when an organization catches the intrusion themselves) averages six to seven days; external notification (a third party telling you that you have a problem) is worse. "You're still kind of failing at your job of detecting and eradicating threats effectively," Craig told the audience at BSidesSF 2025.

The talk draws a through line from John Flynn's 2012 Black Hat presentation "Intrusion Detection Along the Kill Chain" to the present day, updating the core thesis for cloud-native infrastructure, modern threat actors, and the realities of small detection and response teams.

The 80 Percent Problem

▶ Watch: The 80 percent problem — ambiguous events (10:01)

Craig opened with a framing that resonated with anyone who has worked detection: roughly 20 percent of what a D&R team sees is clearly malicious — lateral movement, obvious IOCs, things that demand immediate IR. Another small fraction is obviously benign. But 80 percent of events sit in a grey zone: interesting, suspicious, hard to call without more context.

"Eighty percent of our job as detection people is the work phase — figuring out how to apply work to that eighty percent corpus," Craig said. That work phase is where most teams fail, and where AI tools still fall short. Craig ran a detection-oriented query through an enterprise AI platform during his preparation for the talk and found the output unreliable. "Don't worry about your job yet. Maybe six months, we'll see."

The implication is that until AI can reliably handle that ambiguous middle, human analysts need better tools and workflows — not just more rules.

The Prerequisites: What You Need Before Detection Works

▶ Watch: Prerequisites for effective detection (12:01)

Craig was direct about the organizational and technical prerequisites required before any of his detection techniques can function:

• Detection-as-code with a CI/CD pipeline. If detections are not managed like software — version controlled, tested, deployed through a pipeline — the rest of the framework will not hold. "If you don't have a detection-as-code life, this talk is not for you."
• A structured protocol for interacting with data (queries, analytics), alerting platforms (Slack, Jira, PagerDuty), and response tooling.
• A defensible posture. Craig's minimum bar is enforced FIDO2/WebAuthn for all corporate authentication. "If you're not living the FIDO2 WebAuthn life, good luck. You're gonna have a bad time." He distinguished sharply between hardware-backed FIDO2 (YubiKeys) and passkeys synced to cloud keystores, which he views as materially weaker for enterprise contexts.
• User and machine identity profiles that can be compared to themselves over time and to cohort baselines. A user's home IP, mobile device, browser user agent, and typical login geography are surprisingly stable signals that become powerful anomaly detectors when aggregated.

Craig also emphasized the XDR-style data model: identity logs, EDR logs, and cloud logs must be co-located or at least efficiently cross-queryable. "You need a way to performantly query things."

Enrichment, Decoration, and Cross-Correlation: Three Use Cases

▶ Watch: SSO login use case (18:02)

The heart of the talk was a set of three worked examples showing how raw alerts become actionable intelligence — or confidently ignorable noise — through successive layers of context.

Use case 1: Suspicious SSO login. The raw alert: new login, new device, new location, new IP, flagged as suspicious. The first enrichment step resolves the IP to a Mullvad VPN exit node in Germany, while the user normally logs in from San Francisco. A naive system pages on-call. But one pivot to EDR logs reveals the user's corporate-managed laptop — with an active EDR agent, a valid hardware FIDO2 authentication, and a Chrome process traced back to their login session — is on the same IP. A second pivot shows the user downloaded and installed Mullvad VPN a week prior. The verdict: not worth waking anyone up. The employee should receive a Slack notification about VPN policy, not a 3am incident page.

Use case 2: New admin account created. Jane adds Jason as super-admin on a service. The automation platform queries the HR system (live or via a key-value store cron) to confirm both accounts are active employees with appropriate roles, then checks change management in Jira for a ticket authorizing the elevation. If a valid change ticket exists and both accounts match HR records, the severity drops significantly. Craig noted this scenario also illustrates the value of -admin accounts — separate privileged identities distinct from the accounts used for day-to-day tasks like reading email.

Use case 3: Suspicious AWS GuardDuty finding. GuardDuty fires on a database login it considers unusual. ARN mapping and session token pivoting reveals it was jason-c-admin assuming a service role and accessing an RDS instance assigned to his team. EDR shows a terminal process running from his standard shell, no anomalous child processes, and a GitHub repository on the same branch that might explain why a manager-class account accessed a database. Again: no page needed.

▶ Watch: AWS GuardDuty use case (24:02)

Each case illustrates the same loop: raw alert → enrich with external data (IP reputation, VPN detection, HR data, threat intel) → cross-correlate with adjacent data sources (EDR, SSO, Git, Jira) → decorate with identity and machine profiles → make a decision or take an automated action.

Applying the Framework to Nation-State Actors

▶ Watch: Applying the framework to Lapsus$ (26:02)

Craig applied the same framework to a Lapsus$-style attack scenario, where the assumption is that FIDO2 enforcement and zero-trust device posture are deliberately suspended for the hypothetical. When a threat intel team spots lookalike domains registered ahead of a phishing campaign and feeds those indicators into the D&R platform, the system can auto-deploy blocks before the phishing starts — rather than reacting after credential theft occurs.

He also addressed an advanced EDR tampering scenario: if an attacker is selectively pruning their own activity from EDR telemetry, cohort analysis across peer machines would still expose the gap. "Cross-correlation and cohort analysis and decoration of these types of events would still catch that."

The talk also highlighted canaries — tokens embedded in SSO login pages, documents, and credential stores — as a high-signal, low-noise detection mechanism that catches the lazy web-clone style of phishing campaigns at near-zero false-positive rates.

Response: Give Responders a Rich Payload

The final section addressed the responder's experience. Craig's position is that a detection engineering team's primary obligation when generating an alert is not to flag something interesting — it is to hand the responder a rich, actionable package of information so they can make a decision without logging into five tools.

Automated response actions available to well-instrumented teams include: network containment of hosts via EDR, account rotation and session revocation, MFA revocation, automated forensic collection, cloud account isolation, and IOC population into blocking systems. External enrichment sources Craig mentioned (without endorsing specifically) include Census, DomainTools, URLScan, and Spur.

The closing summary was concise: "Context, decoration, enrichment, and cross-correlation take isolated atomic events and turn those into actionable insights where you can actually do something."

Notable Quotes

"False negatives are challenging — and false negatives are like an after-action KPI you can try to figure out. But if you're not thinking about false negatives and you work in the detection space, you really should think about it harder." — Jason Craig (06:01)

"If you just take your SSO logs, you're gonna get paged at 3:00 in the morning for this every single time." — Jason Craig (20:02)

"Context, decoration, enrichment, and cross-correlation take isolated atomic events and turn those into actionable insights where you can actually do something." — Jason Craig (30:03)

Key Takeaways

• Median breach dwell time is still 10–15 days. The data from DBIR and M-Trends 2025 confirms the detection problem has not been solved.
• 80 percent of detected events are ambiguous and require active analytical work. Raw alert volume is not the bottleneck — context is.
• Detection-as-code, FIDO2 enforcement, and co-located identity/EDR/cloud logs are baseline prerequisites before advanced detection techniques pay off.
• User and machine profiles over time are among the most powerful anomaly detectors available — and they do not require expensive tooling, just engineering discipline.
• Enrich, decorate, and cross-correlate before alerting. The SSO login that looks alarming in isolation becomes obvious when viewed alongside EDR, HR data, and a Jira change ticket.
• Hardware-backed FIDO2 (YubiKeys) and passkeys have different enterprise threat models. Enforce the right one and log which authenticator type is used for every session.