Rethinking how we evaluate security agents for real-world use

Rethinking How We Evaluate Security Agents for Real-World Use

Speaker: Mudita Khurana, Staff Security Engineer, Airbnb

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 4, 2026, San Francisco

Watch: YouTube — Full Talk

Reading time: ~5 min

TL;DR

An 80% benchmark score on a security agent tells you almost nothing useful. Mudita Khurana's lightning talk introduces CLASP, a capability-centric evaluation framework that shifts the question from "did the agent succeed?" to "how did the agent succeed?" — enabling meaningful debugging, targeted improvement, and reliable deployment decisions.

Introduction

At [un]prompted 2026, Mudita Khurana — a Staff Security Engineer at Airbnb presenting personal research — opened with a deceptively simple observation: we've all met someone who gives the right answer to a question but clearly doesn't understand why it's right. You can't predict whether they'll get it right next time. You can't give them useful feedback because there's no underlying skill to improve.

The same problem applies to AI agents evaluated on outcome-only benchmarks. An agent that scores 80% on a security benchmark might be doing shallow pattern-matching all the way to the correct answer — and you'd have no way of knowing. "That doesn't tell you why it worked 80% of the time. It doesn't tell you why it failed the other 20%. It doesn't tell you how to improve that 80% to 90, 95, or closer to 100."

This is the motivation behind CLASP.

▶ Watch: The motivation (00:00)

The Problem: Outcome Success Is Not Real-World Success

Khurana illustrated the failure mode with a concrete SQL injection example. An agent examining a vulnerable code endpoint outputs: "This endpoint contains a SQL injection vulnerability. The user_id parameter is interpolated directly into the query string without parameterization or sanitization." The outcome score: correct. Pass.

But looking at the agent's actual trace tells a different story. The agent executed three steps: detected an f-string with a variable inside a SQL query, matched it to a known SQLi template pattern, output the finding. What it did not do: trace the taint flow from the request through to the query sink, check whether the data was sanitized downstream by an ORM wrapper, or produce any handoff artifact with reproduction steps or a payload idea.

The agent did a superficial analysis. In an isolated benchmark, it passed. In a real security workflow, this failure cascades. Security is not a series of isolated tasks — it's a closed loop: find the bug, confirm exploit, patch, validate. An agent that found the bug via shallow heuristics will not produce the evidence needed for the exploit confirmation step. What tends to happen: incomplete patches get written, the end-to-end workflow breaks, and the vulnerability persists.

▶ Watch: The SQL injection example (02:00)

CLASP: Six Capabilities, Five Levels

CLASP (Capability-centric evaluation framework) evaluates agents across six agentic capabilities, each scored on a 1–5 scale from brittle to adaptive:

| Capability | Key Question |

|---|---|

| Planning | Did it have a strategy, or just fire tools? |

| Tool Use | Did it pick the right tool and use it correctly? |

| Memory | Did the details survive to the next step? |

| Reasoning | Did it analyze, or pattern-match? |

| Reflection | Did it check its own work? |

| Perception | Did it read the full output, or just the obvious part? |

A score of 1 means minimal, brittle execution. A score of 5 means adaptive, self-correcting execution. The framework pushes developers to ask "how did we achieve this success?" over and above "did we achieve this success?"

One of the more actionable findings from applying CLASP to existing agents: not all capabilities matter equally for all security tasks. For enumeration-heavy tasks like reconnaissance, breadth matters more than depth — agents that planned which tools to use and enumerated extensively outperformed agents that reasoned deeply about individual findings. Different security workflow stages have different capability bottlenecks, and CLASP's skill attribution table maps which capabilities to prioritize for each stage.

▶ Watch: CLASP framework and skill attribution (04:00)

Using CLASP in Practice: A Three-Step Approach

Khurana outlined a practical three-step workflow for applying CLASP:

Step 1 — Observability. Instrument your agent workflows to capture everything: plans and subgoals, reasoning traces, tool calls and their outputs, retrieved context. You need the full behavioral trail to evaluate anything meaningfully.

Step 2 — Evaluate. Two options depending on your investment appetite. The lighter lift: build an LLM-as-a-judge pipeline. Feed it the CLASP rubric and your agent's trace, ask it to grade planning and tool use. A score of 2 on the rubric tells you exactly what you need to do to get to a 4. The heavier lift: build test scenarios for each capability at varying complexity levels, using evidence-centered benchmark design. A low tool-use score looks like an agent cycling through a static set of tools in the same order with no environmental feedback. A high tool-use score looks like an agent chaining tools dynamically, listening to errors, and constructing its approach adaptively.

Step 3 — Improve and gate. Use the capability-to-task mapping table to know what to work on first. Ship only when both outcome success AND minimum capability thresholds are met — then regression-test on every change.

▶ Watch: CLASP in practice (06:00)

Notable Quotes

"Agent evaluation today is rooted in very narrow outcome-only scoring, which essentially hides explainability and reliability." — ▶ 00:00

"Capability-centric optimization buys you reliability and parsimony — target the bottleneck skills and you get fewer tokens, fewer tool calls, fewer retries, less human review, lower cost for the same or better success." — ▶ 08:00

"Don't evaluate on isolated narrow outcomes. Dig deeper into the how of these agents." — ▶ 08:00

Key Takeaways

• Outcome metrics alone are necessary but not sufficient. An agent can score well on a benchmark via shallow pattern-matching and fail catastrophically in a real workflow where it needs to carry context forward.
• Security is a closed-loop workflow. Evaluating individual stages in isolation misses how context loss between stages causes end-to-end failures even when each stage looks like a pass.
• CLASP provides a structured path from diagnosis to improvement. If your agent scores a 2 on planning, the rubric tells you specifically what behaviors to develop to reach a 4.
• Different security tasks need different capabilities. For recon, breadth beats depth. For exploitation, reasoning quality matters more. CLASP's skill attribution table gives you a starting point for where to invest.
• Gate on capability thresholds, not just outcomes. Shipping when outcome success AND minimum CLASP scores are met — then regression-testing on every change — catches degradation that pure outcome monitoring misses.

Slides and Paper Reference

Slides are available from the conference as 2026-04-04-D2-S1-12-09-Rethinking-how-we-evaluate-security-ag.pdf. The full academic paper (Khurana's personal research, not affiliated with Airbnb) is referenced in the talk and contains the full CLASP rubric tables, skill attribution by security task stage, and a blueprint for the benchmark the team is actively developing.