The Hard Part Isn't Building the Agent: On Measuring Agent Effectiveness to Improve It

The Hard Part Isn't Building the Agent: On Measuring Agent Effectiveness to Improve It

Speaker: Joshua Saxe, AI Security Technical Lead (formerly Meta); Co-founder, new startup

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 3–4, 2026, San Francisco

Watch on YouTube: https://www.youtube.com/watch?v=rO2yA52U_i4

Reading time: ~9 minutes

TL;DR

Joshua Saxe makes a counterintuitive argument: the biggest blocker to deploying autonomous AI security agents isn't building them — it's evaluating them. Classical ML metrics like precision, recall, and F-score fail in cybersecurity because the ground truth is structurally noisy. The solution is to treat AI agents the way you'd evaluate a human security engineer: assess the quality of their reasoning process, not just their binary outputs.

Introduction

Joshua Saxe was, as of the week of the conference, newly departed from Meta, where for four years he led AI security work across the company — building systems that applied AI to the full range of security problems at Meta scale, including prompt injection defense and defense against AI-native attacks. He showed up at [un]prompted 2026 to deliver a talk he acknowledged was unusual for an opening slot: a careful, technically grounded argument about statistical evaluation methodology.

His thesis is that the security industry has imported classical machine learning evaluation tools into a domain where they fundamentally do not apply, and that this mismatch is the primary reason AI security teams fail to ship reliable autonomous systems. Getting evaluation right, he argued, is the prerequisite for everything else.

▶ Watch: Introduction and Background (00:00)

Why Autonomous Defenders Are Necessary Now

Saxe opened by amplifying the threat picture described in the preceding talk. The trajectory is not gradual: attackers are transitioning from being manual laborers who conduct attacks step-by-step to being managers of AI agents that conduct attacks at scale. This means defenders need their own autonomous systems.

The practical stakes are immediate. Many organizations have very few IT security staff — a hospital with two IT security workers cannot cover its entire threat surface. In that context, AI agents will need to be trusted to quarantine a compromised executive's account, patch a production codebase, and shut down compromised hosts — entirely on their own, without human review of each decision.

▶ Watch: The Case for Autonomous Defense (00:00)

This creates an uncomfortable problem: how do you trust a system making high-stakes autonomous decisions when you cannot fully understand the neural network underlying it? Mechanistic interpretability — staring into weight matrices to verify behavior — is not a practical path forward. The only real option is to understand these systems statistically and behaviorally. Which means evaluation is the hard problem.

The Ground Truth Problem

Saxe spent the middle section of his talk on what he called "the bad news": the data that AI security systems are evaluated against is fundamentally, structurally noisy — and the field has been pretending otherwise.

Classical machine learning metrics were built for problems with transparent ground truth. The canonical example: is there a cat or a dog in this image? The label is unambiguous. Security is not that kind of problem.

▶ Watch: The Noise Problem in Security Ground Truth (04:00)

Consider three concrete examples Saxe has encountered personally:

SOC alert triage. If you try to get ground truth labels on which SOC alerts are true positives versus false positives, you find a double-digit disagreement rate between experienced SOC analysts. The labels are deeply noisy, and yet teams evaluate AI triage systems against them as if they were authoritative.

Access management. When determining who should have access to a sensitive database table containing user messages, access reviewers disagree at double-digit rates. An AI system trained or evaluated against those labels inherits that disagreement as measurement error.

Malware classification. Many binaries are genuinely dual-use. The question of whether a program is malware involves ontological and epistemological problems that cannot be resolved by labeling. There is no oracle.

Saxe ran a simulation to quantify what this noise does to measurements. A perfect AI system that correctly predicts whether a SOC alert is a false positive starts to look inaccurate as soon as any label noise is introduced. At 1% to 3% label noise — a range that is essentially guaranteed in cybersecurity, not a worst-case assumption — you hit a "noise ceiling" above which it becomes impossible to measure whether the system is improving at all.

▶ Watch: The Noise Ceiling Simulation (08:00)

The structural problems are permanent. The halting problem and state-space explosion make it impossible to know with certainty whether a program is bug-free or whether a patch will crash production. Societal disagreements about dual-use binaries will not be resolved. "We will always be operating in this uncertain environment."

The Framing Problem: Oracle Versus Decision-Maker

The current evaluation model, Saxe argued, rests on a flawed assumption: that somewhere there exists an oracle who can look at a security artifact and label it correctly. Against that imagined oracle, we compare the agent's binary outputs — true positive or false positive — and compute precision, recall, F-score, ROC curves. We treat noisy labels as ground truth. And we reduce everything the agent actually did — potentially 100,000 tokens of reasoning, tool calls, evidence gathering, explanatory narrative — down to a single bit of comparison.

"Imagine how you would feel as an AI agent if that was how your performance was evaluated."

The better analogy, Saxe argued, is hiring. When a company hires a security engineer, it doesn't evaluate them purely through binary multiple-choice tests. It assesses how they think under uncertainty. It asks about their reasoning process, their evidence gathering, their ability to explain their decisions clearly and justify their choices with first principles.

▶ Watch: Reframing Evaluation as Hiring (12:01)

The thesis: classical metrics are important and should not be abandoned, but they need to be paired with systematic evaluation of how agents reason and behave. "Our mental model should be the same as when we hire a security engineer — we are assessing the ability to make good, well-validated decisions under the extreme uncertainty of this field."

A Practical Approach: Rubric-Based Agent Evaluation

Saxe described the approach he has operationalized in his own work:

Define an interview rubric. For a given task — say, an access management decision about whether to grant a contractor access to a database table containing user messages — define what good performance looks like across multiple dimensions: Did the agent gather the right evidence? Did it reason correctly from first principles? Did it justify its decision clearly? Did it produce an auditable trail? Did it arrive at the same conclusion as your best security engineers in comparable past incidents?

Grade trajectories at scale. Score complete agent trajectories — not just outputs — against the rubric. Because doing this manually doesn't scale, train an LLM judge to automate the grading. Saxe noted that you need far fewer samples than expected: often 100 labeled examples is enough to calibrate a judge and get started.

Define a deployment bar and hill-climb. Set a threshold across all holistic dimensions that leadership has aligned on. Iterate the agent until it exceeds that threshold. Deploy. Monitor for drift from steady state.

▶ Watch: Practical Rubric-Based Evaluation (14:02)

The contrast Saxe drew was stark: in the "dystopian" version, the team argues endlessly about whether poor metrics mean the model is wrong or the labels are wrong (usually the latter). In the better version, many dimensions are measured simultaneously, all somewhat noisy but together providing a much more complete picture of whether the system is working.

One practical note Saxe emphasized: many AI security teams right now are operating with no evaluation program whatsoever. "This is a genuine crisis — because then it is all vibes. Engineers tweaking their agents and saying it works on their machine, with no reproducible bar for shipping." An imperfect evaluation program is far better than no program.

Time and Resource Investment

Saxe was direct about cost: he estimates that at least 50% of a team's engineering time should go to evaluation. The payoff is roughly a tenfold increase in shipping velocity because the team has a reliable signal instead of noise. When leadership has aligned on the deployment bar, reaching it becomes a clear engineering objective rather than an open-ended argument.

He also briefly noted a downstream benefit of strong evaluation: once an eval system is in place, AI coding tools and genetic-algorithm-based prompt optimization can automatically hill-climb agent performance — but only if the measurement system is trustworthy enough to tell the difference between real improvement and overfitting to noise.

▶ Watch: Investment and Timeline Pressures (18:02)

Q&A Highlights

An audience member asked about improving LLM judges by incorporating negative feedback from human raters who disagree with judge decisions. Saxe's answer was essentially yes: the right model involves weighting labels by inter-rater agreement. Labels where three reviewers strongly agree get weighted more highly than labels where reviewers disagree — something like a Bayesian calibration model over the evaluation data. A second questioner asked whether security-specific evaluation differs materially from evaluation methods in other AI domains. Saxe's position: the statistics themselves (multi-dimensional rubric scoring, inter-rater calibration) are standard; the domain-specific challenge is that the ground truth is uniquely noisy in ways that make the multi-dimensional approach not just preferable but necessary.

Notable Quotes

"Evaluation is the single biggest blocker in our field." — Joshua Saxe (04:00)

"We have reduced everything the agent did — potentially 100,000 tokens of reasoning, tool calls, evidence gathering — down to a single bit of comparison against a noisy label. Imagine how you would feel as an AI agent if that was how your performance was evaluated." — Joshua Saxe (10:00)

"Many AI security teams right now are operating with no evaluation program whatsoever. It is all vibes — engineers tweaking their agents and saying it works on their machine." — Joshua Saxe (16:02)

"It sounds like science fiction, but it is quite reasonable to expect armies of artificially general intelligent robots scaling up adversarial activity against all of our networks." — Joshua Saxe (18:02)

Key Takeaways

• Classical ML metrics fail in cybersecurity because the ground truth is structurally noisy — SOC analysts disagree at double-digit rates, malware classification is inherently ambiguous, and the halting problem makes patch correctness unknowable with certainty.
• A 1–3% label noise rate creates a measurement ceiling above which it is impossible to tell whether an AI security system is improving — and this noise rate is essentially guaranteed in cybersecurity, not a worst-case scenario.
• Treat AI agents like human hires. Evaluate the quality of their reasoning process across multiple rubric dimensions — evidence gathering, first-principles reasoning, decision justification, auditability — not just their binary output accuracy.
• LLM judges calibrated on ~100 labeled examples can automate rubric-based evaluation at scale, making holistic multi-dimensional evaluation tractable.
• At least 50% of team time should go to evaluation. The return is roughly a 10x increase in shipping velocity because the signal becomes reliable rather than arguable.

Slides Reference

Slides from this session are available as 2026-04-04-D1-S1-9-39-The-Hard-Part-Isn-t-Building-the-Agent-.pdf. Key topics in the slides include: the noise ceiling simulation, the oracle-vs-decision-maker framing comparison, the access management rubric example (evidence gathering, policy understanding, first-principles reasoning, auditability, decision accuracy), and the deployment bar hill-climbing workflow.