1.8M Prompts, 30 Alerts: Hunting Abuse in a User-Defined Agent Ecosystem

1.8M Prompts, 30 Alerts: Hunting Abuse in a User-Defined Agent Ecosystem

Speakers: Matt Rittinghouse (Lead Security Data Scientist, Salesforce); Millie Huang (Staff Security Data Scientist, Salesforce)

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 4, 2026, San Francisco

Watch on YouTube: https://www.youtube.com/watch?v=PtWwrOm3BeE

Reading time: ~9 minutes

TL;DR

Salesforce's security data science team built a behavioral anomaly detection system that filters 1.8 million daily agent prompts down to fewer than 30 actionable alerts — without ever reading customer prompt data. The key insight: stop policing what agents say and start watching what they do at the execution layer.

Introduction

When an AI agent goes rogue inside a major enterprise platform, the threat looks nothing like a traditional cyberattack. There's no malware to signature-match, no suspicious login to flag. Instead, a legitimate customer service agent quietly begins pulling far more records than it should — each individual query perfectly valid, the cumulative pattern deeply wrong.

This is the challenge Matt Rittinghouse and Millie Huang faced as security data scientists embedded in Salesforce's Cybersecurity Operations Center. Their talk at [un]prompted 2026 documented what may be the most mature production deployment of AI agent behavioral monitoring in the industry: a system covering 55,000 organizations and 12,000 unique daily active agents, distilling nearly two million daily prompts into a handful of high-confidence alerts.

The result reframes how the security community should think about agentic threats — shifting from content-layer defenses to execution-layer behavioral analysis.

The Wild West Problem: Why Agents Break Traditional Security

▶ Watch: The shift in threat surface (00:00)

Rittinghouse opened with a framing that resonated throughout the room: "We are now officially past the chatbot era." The threat model has fundamentally changed. When an agent can autonomously call APIs, query databases, execute code, and manipulate infrastructure, security stops being about filtering bad words and starts being about monitoring authorized system execution.

Salesforce's Agentforce platform deploys agents across both its own internal organizations and customer tenants — a multi-tenant SaaS environment of enormous complexity. Every agent is custom-built by the organization that deploys it. Valid behaviors vary wildly between a customer service bot for a healthcare company and a sales automation agent for a financial services firm.

This creates what the speakers called the "black box defense challenge": static signatures simply cannot scale across 12,000 unique daily active agents. Every rule you write for one agent breaks for another.

The threat landscape splits into two buckets. Platform-targeted attacks exploit implementation flaws — misconfigured permission sets or vulnerabilities in custom skills exposed to the reasoning engine. Abuse of legitimate agency is subtler and more dangerous: threat actors using an agent's built-in capabilities to perform individually valid actions that are collectively malicious. Data reconnaissance through a support agent. Exfiltration disguised as normal record retrieval.

"We aren't just defending against bad words," Rittinghouse said. "We're defending a massive multi-tenant execution engine where legitimate logic is constantly evolving."

Why Content Moderation Alone Isn't Enough

▶ Watch: The observability and enforcement gap (02:00)

Salesforce already had a content moderation layer — what the team calls the "trust layer" — that screens the dialogue between users and the LLM for toxicity and prompt injection. It is, they emphasized, critical infrastructure. But it has three blind spots that make it insufficient as a standalone defense for an agentic system.

Reasoning vs. execution. The trust layer polices the thought process — the conversation between user and model. It cannot see the actual system calls the agent makes after reasoning concludes. An agent can produce a perfectly safe-looking reasoning trace and then execute a malicious database sweep.

The blocking dilemma. Content moderation can surface suspicious intent, but it lacks the high-fidelity confidence required to trigger automated inline blocking in an enterprise context. False positives carry real business cost. You need something more precise before you can automate containment.

Post-generation blindness. Even a prompt deemed safe can produce an agentic plan involving unauthorized data access or privilege escalation — steps that unfold after the content filter has already cleared the request.

"Content moderation alone is not enough," Huang stated plainly. The behavioral detection layer the team built sits outside but works in concert with the trust layer, operating at the execution layer where the actual damage occurs.

The Ensemble Model: Three Axes of Behavioral Context

▶ Watch: Building the anomaly detection model (06:00)

The core innovation is conceptually elegant: treat the agent itself as a new axis for anomaly detection, in addition to the user and the organization. This three-layer context — user, agent, organization — forms the basis of an ensemble model that surfaces deviations that any single layer would miss.

The hypothesis was deliberately simple: "agents accessing things they normally don't access" is a reliable proxy for anomalous intent. Keeping the hypothesis narrow let the team iterate on coverage gaps without drowning in complexity.

The key features cluster into two categories:

Depth of access: Frequency of database calls, proportion of total data assets accessed in a session. A support agent pulling 13 distinct Salesforce objects (Case, ContentDocumentLink, EmailMessage, and others) in a one-minute session is doing something fundamentally different from its historical baseline.

Sensitivity and rarity: Salesforce's data schema includes feature flags marking fields containing PII or sensitive data. The model profiles how frequently agents touch certain columns and treats rare access patterns as elevated signals. The example shown in the slides: a session that scored 5.65 standard deviations above the user's typical behavior — 581 previous sessions worth of historical context making the deviation unmistakably significant.

Incremental historical profiling ties these dimensions together. Every day, a profile of all sessions is assembled. As the rolling history grows, standard deviation-based scoring becomes more precise. Critically, this lets the team score sessions in near-real time — even as a session is still ongoing — and it bakes alert prioritization directly into the model output.

▶ Watch: What the features look like and what didn't work (08:00)

What Didn't Work: Hard-Won Engineering Lessons

Rittinghouse was candid about the dead ends. The team's early model had over 100 features — a natural consequence of multiplying data rarity and frequency signals across three identity layers (user, agent, org). "It is incredibly easy to explode features when you're working with multiple levels of identity," he said.

The most instructive failure was a custom query complexity calculator. The hypothesis seemed reasonable: malicious actors would run complex, unusual queries. What the team discovered was that query complexity is inherent to the agent's design, not to any malicious intent. Agents themselves construct queries; their complexity reflects how the agent was built, not what the user is trying to do. "Make sure your features are measuring what you think they are" became the lesson.

A second engineering pain point: pulling contextual metadata from multiple tables required expensive database joins that significantly slowed training. After refactoring to operate on a single table and removing those joins, training time dropped to one-third of its previous duration — a critical enabler for moving toward real-time scoring.

▶ Watch: Results and the agentic triage loop (12:01)

The Results: 1.8M to 30

The headline numbers are striking on their own. Nearly two million daily prompts. Trust-layer filtering removes some fraction. The behavioral anomaly detection model then compresses the remainder to fewer than 30 active alerts requiring human investigation.

But the team didn't stop at detection. They built an agentic triage layer: each alert generates a structured JSON payload ingested by a downstream agent that synthesizes a plain-English summary of why the session is anomalous. The output is customer-facing and accessible to investigators who may not have a deep data science background — describing the deviation in terms of what the agent did relative to its history, not raw statistical scores.

The Road Ahead: Auto-Containment at the Execution Layer

▶ Watch: The path to real-time auto-containment (14:01)

The current pilot operates with 12-to-24-hour detection latency, since training and inference both run in batch jobs. The architecture is already designed for the hot path. By pushing behavioral baselines learned during daily batch processing to a high-speed cache, the team can perform inference in-flight as sessions occur.

When a session hits a critical threshold — statistically near-impossible for the behavior to be normal — the system moves to auto-containment: kill the session, revoke the token, or trigger a bot-level lockdown, without waiting for a SOC analyst to triage the alert. "That is how we will close the loop at the execution layer of an agentic system," Huang said.

Notable Quotes

"We aren't just defending against bad words. We're defending a massive multi-tenant execution engine where legitimate logic is constantly evolving."

— Matt Rittinghouse (00:04)

"Without that context, you're blind to the abuse-of-legitimate-agency case and cannot distinguish a power user from a compromised agent."

— Millie Huang (14:01)

"Simple telemetry — invocation count, sensitive asset frequency — produced way higher fidelity signals without having to blow up our SLAs."

— Matt Rittinghouse (16:01)

"Do expect noise for the first fourteen days of a new agent's life. Build that warm-up period into your SOC's playbook, so you don't burn out your analysts on day one."

— Millie Huang (16:01)

Key Takeaways

• Execution layer defense is non-negotiable. Content moderation cannot see system calls. You need a separate detection layer that monitors what agents do, not just what they say.
• Agent identity is a new axis for anomaly detection. In agentic ecosystems, profiling at user + agent + org creates an ensemble that no single-layer approach can replicate.
• Keep your hypothesis and features simple. Complexity explodes fast when working across multiple identity layers. Invocation count and sensitive asset frequency outperformed a sophisticated query complexity model.
• Log structured events, not debug strings. If your agentic logs don't link invoking user ID to agent ID, you are blind to the abuse-of-legitimate-agency case from the start.
• Plan for a 14-day warm-up period. New agents need time to establish a behavioral baseline. Build this into your SOC playbook or you will bury analysts in false positives on day one.

Slides Reference

Slides available: 1.8M Prompts, 30 Alerts: Hunting Abuse in a User-Defined Agent Ecosystem (Salesforce Cybersecurity Operations Center). Key slide topics include: the two-pronged threat model (platform attacks vs. abuse of legitimate agency), the observability and enforcement gap, the "wisdom of the crowd" ensemble approach, feature overview (call frequency, asset proportion, PII access, data rarity), incremental historical profiling methodology, engineering trade-offs (expensive joins eliminated), results summary, and a real-world alert example showing a 5.65 standard deviation anomaly score across 581 historical sessions.