United in Defense: Architecting Safe and Trustworthy AI Agents

United in Defense: Architecting Safe and Trustworthy AI Agents

Speakers: Rabimbo, Google Developer Expert and Scientist at PayPal

Conference: BSides Seattle

YouTube: https://www.youtube.com/watch?v=Yy9VFRMwe98

Overview

Rabimbo, a Google Developer Expert and scientist at PayPal, presents a comprehensive security framework for AI agent systems, covering the full lifecycle from training data provenance through model serving to multi-agent output validation. Unlike many AI security talks that focus narrowly on prompt injection, Rabimbo maps the broader threat surface including data poisoning, model source tampering, backdoored neural networks, pickle deserialization attacks, bearer token exposure, sensitive data inference, and the cascading failure modes of multi-agent architectures.

The talk draws from Rabimbo's hands-on experience building and securing LLM-based systems at PayPal, where the team discovered that even models restricted to code-only output could be manipulated into producing code that, when chained, disclosed sensitive information the model should never have been able to access. This real-world finding underscores the talk's central message: securing AI agents is a "running goalpost" where no single defensive layer is sufficient.

Framed around Google's Secure AI Framework (SAIF), the presentation is intentionally model-agnostic and framework-agnostic, addressing the reality that the AI ecosystem evolves too rapidly for security guidance tied to specific tools or versions.

Background

▶ Watch: Rabimbo introduces AI agent security concerns from PayPal experience (0:15)

The explosive growth of AI agent deployment has outpaced security practices. As Rabimbo observes, every day brings new models, new frameworks, and new agent architectures -- OpenClaw gets renamed before you can download it, and by the time Cloudflare ships a model runner, the underlying model has already changed. Google's Secure AI Framework (SAIF) was released as a comprehensive guide, but practitioners found that simply following it did not solve real-world problems because the pace of change made static frameworks insufficient.

The talk maps the complete AI system attack surface across four layers: the model layer (the LLM itself and its training data), the infrastructure layer (where models are stored, trained, and served), the application layer (how users interact with the model, including input/output handling), and the governance and assurance layer (which Rabimbo notes frequently comes as an afterthought, creating downstream problems).

HuggingFace conducted a study that found a significant number of hosted models had been intentionally tampered with, confirming that model supply chain attacks are not theoretical but actively occurring in the wild.

Key Findings

▶ Watch: Data poisoning attack: training classifiers with coordinated false reports (5:45)

Pickle deserialization as a model delivery attack: Python's pickle serialization format, used extensively for saving and loading ML models, is inherently exploitable through its __reduce__ method. Rabimbo demonstrates a poisoned model file that, when loaded by an unsuspecting developer, opens a reverse shell to the attacker. The demo showed this working against a 30 MB ONNX model distributed via HuggingFace, and at the time of testing (six to seven months prior), Windows Defender did not flag the malicious file.

Backdoored neural networks: Models can be trained to respond normally to standard inputs but execute entirely different behavior when specific trigger patterns are present. In multi-agent systems, this is particularly dangerous because a backdoored model loaded by one agent can instruct that agent to "forget some of its guards," cascading failures through the entire agent chain.

Inferred sensitive data disclosure: Perhaps the most striking finding from Rabimbo's experience at PayPal. A model restricted to code-only output was manipulated by users who asked it to produce specific code that, when executed as a chain, revealed sensitive information the model was never intended to output. Because the model only produces code, the team had not implemented private data disclosure controls -- they assumed code output was inherently safe. This assumption proved wrong.

Multi-agent cascading failures: In multi-agent architectures where agents have different responsibilities and permission scopes, a compromise of one agent (through any of the attack vectors discussed) can cascade through the system. Rabimbo's team found that traditional sandboxing was "not enough a lot of times" to contain these failures.

Technical Deep Dive

▶ Watch: Demo: pickle deserialization attack opens reverse shell via poisoned ONNX model (10:30)

Rabimbo breaks the defensive architecture into several interconnected components:

Data provenance and sanitization: For organizations fine-tuning models or retraining on proprietary data, the training data pipeline requires explicit sanitization. Rabimbo describes a scenario where a log-analysis AI model fine-tuned on example datasets from HuggingFace could inherit poisoned patterns without the team's knowledge. A more sophisticated attack involves creating coordinated cloud bots to systematically report false positives/negatives to abuse user-reporting flows (e.g., Gmail spam classification), training classifiers to learn patterns they should not.

Model source verification: The pickle deserialization attack exploits Python's __reduce__ method to execute arbitrary code during model loading. The defense requires implementing verifiable provenance using cryptographic signatures for model artifacts and treating model files with the same suspicion as executable code. The ONNX format was used in the demo, showing the attack works across common ML serialization formats.

Access control at the agent level: Each agent in a multi-agent system needs its own scoped authentication, separate from other agents. Rabimbo's team at PayPal implements per-agent scoping where each agent has isolated credentials and permission boundaries. The critical question is whether an agent inherits the user's permission scope or has a system-level scope -- and whether the agent can be manipulated into accessing data intended for a different permission level.

RAG access control: For systems using Retrieval-Augmented Generation, ensuring that retrieved documents respect the requesting user's access level is a separate challenge from model access control. An intern querying a RAG-backed documentation system should not receive the same results as a senior engineer, even if both use the same model.

Input/output security classifiers: Rabimbo's team deploys dedicated classification systems that inspect input before it reaches any LLM and validate output before it reaches the user. These are not just prompt injection detectors -- they look for invisible image content designed to hijack accuracy, tampered logs that suppress results when processed by log analysis systems, and code output that could chain into sensitive data disclosure. The team learned that sandboxing alone was insufficient and that dedicated, purpose-built classifiers were necessary.

Differential privacy training: For models trained on sensitive data (healthcare, financial), differential privacy provides mathematical guarantees that individual data points cannot be extracted from model outputs. Rabimbo notes this is essential because empirical testing can show a model does not disclose private data, but cannot guarantee it. Differential privacy provides that guarantee, though it comes with some utility cost.

Evaluation beyond accuracy: Model evaluations must capture not just what the model should output but explicitly what it should not output. Rabimbo's team made a "huge push" to define negative evaluation criteria, testing specifically for outputs the model must never produce rather than only validating correct outputs.

Demo / Proof of Concept

▶ Watch: Backdoored neural networks causing cascading failures in multi-agent systems (14:00)

Rabimbo demonstrated the pickle deserialization attack using two side-by-side VMs. The attacker VM hosted a listener while the victim VM loaded a poisoned ONNX model through a simple Gradio chatbot interface. Upon loading the model, a reverse shell immediately opened on the attacker's VM, providing full file system access to the victim machine. The poisoned model was approximately 30 MB and available on GitHub and HuggingFace for reproduction (with appropriate warnings to run only in isolated VMs). Windows Defender did not detect the malicious payload at the time of testing.

Defensive Implications

▶ Watch: Inferred sensitive data: code-only model leaks info through chained execution (22:00)

For organizations deploying AI agents, Rabimbo's practical recommendations center on defense-in-depth across the entire AI lifecycle:

Review AI workflows end-to-end, from the data science team through implementation to data providers. The audit trail becomes exponentially more complex as multi-agent systems scale, and most organizations underinvest in AI-specific auditing controls.

Implement per-agent access control with isolated authentication in multi-agent systems. Agents should not share credentials or inherit overly broad system-level scopes.

Deploy dedicated input and output security classifiers rather than relying solely on model-level guardrails. Prompt injection is "a very small part of the equation" -- more problematic are invisible image content, tampered logs, and chained code outputs.

Adopt differential privacy for training on sensitive data to provide mathematical guarantees beyond empirical testing. Define negative evaluation criteria explicitly -- test for what the model must not produce, not just what it should.

Rabimbo provides a QR code to a GitHub repository containing a lightweight end-to-end implementation of the access controls, sandboxing, and output filtering discussed in the talk.

Key Takeaways

• Pickle deserialization attacks through poisoned ML models can provide reverse shell access, and Windows Defender did not flag the payload at time of testing
• HuggingFace found significant numbers of intentionally tampered models in its repository, confirming model supply chain attacks are active
• Multi-agent systems create cascading failure modes where one compromised agent can undermine the entire chain, and sandboxing alone is insufficient
• Models restricted to code-only output can still disclose sensitive data through chained code execution -- an output format restriction is not a security boundary
• Per-agent scoped authentication and isolated credentials are essential for multi-agent architectures
• Differential privacy provides mathematical guarantees that empirical testing cannot, particularly for models trained on healthcare or financial data
• Define negative evaluation criteria: explicitly test for outputs the model must never produce

About the Speaker(s)

Rabimbo is a Google Developer Expert and scientist at PayPal, where he works on securing LLM-based systems in production. His experience spans end-to-end AI deployment scenarios, and he brings practical perspective from dealing with real-world attacks against multi-agent systems handling sensitive financial data. He maintains a GitHub repository with reference implementations of the security patterns discussed in the talk.