Security for AI Agents Using an Ensemble of Fine-tuned Small Language Models

Security for AI Agents Using an Ensemble of Fine-tuned Small Language Models

Speakers: Lidan Hazout; Bar Kaduri

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=BEvFbJQXDs

Overview

The rapid adoption of AI agents across various industries, from coding assistants to personal productivity tools, has introduced a new and complex attack surface that traditional security paradigms are ill-equipped to handle. This talk, presented by Bar Kaduri at BSides SF, delves into the critical need for robust security mechanisms for these autonomous entities. Kaduri, alongside Lidan Hazout, whose contributions were acknowledged as central to the innovation, outlines a novel runtime security architecture designed to prevent AI agents from executing unintended or malicious actions.

The presentation highlights the inherent unpredictability and non-deterministic nature of AI agents as a core challenge, necessitating a shift towards AI-driven security at runtime. The proposed solution is a multi-layered security gate that intercepts agent intents before execution, leveraging an ensemble of specialized, fine-tuned Small Language Models (SLMs) and an adaptive, Retrieval Augmented Generation (RAG) based memory system. This approach aims to provide real-time, contextual, and knowledge-driven decisions, enabling organizations to deploy AI agents in production environments with confidence and significantly mitigate risks such as agent goal hijack and rogue agent behaviors.

Background

▶ Watch: The core problem: AI agents doing unexpected things. (1:40)

The proliferation of AI agents has ushered in an era where software not only executes predefined instructions but also makes autonomous decisions, interacts with tools, and manipulates data based on complex reasoning. While transformative, this autonomy presents significant security vulnerabilities. The speaker referenced the OWASP Top 10 AI Security AI project (released December 2023) as a crucial community effort to map these emerging risks, underscoring the collaborative need to secure the AI ecosystem.

Two specific risks from the OWASP Top 10 were highlighted as central to the problem addressed:

• ASI 1: Agent Goal Hijack: This refers to scenarios where an agent is tricked into performing actions outside its intended purpose, often through prompt injection or other adversarial inputs. The agent's legitimate instruction set is overridden by a malicious one.
• ASI 10: Rogue Agent: This is perhaps the most insidious risk, as it describes situations where an agent, without any explicit malicious instruction or compromise, autonomously decides to perform an unintended or harmful action. A notable incident cited was a Replit agent inadvertently deleting a production database, a classic "whoops moment" that exemplifies the unpredictability of rogue agents.

Kaduri emphasized that many of these risks stem from how agents interact with their "hands" – the tools and actions they are empowered to take. Traditional application security measures are insufficient because AI agents operate with a level of abstraction and decision-making that is fundamentally different from conventional software. The solution, therefore, must be dynamic, contextual, and capable of operating at the point where an agent transitions from "thinking" to "acting." This realization led to the development of a runtime security approach that tackles the problem at the critical juncture of tool invocation.

Key Findings

▶ Watch: Overview of the OWASP Top 10 AI Security project. (3:10)

The central finding of this research is that a dynamic, multi-faceted security gate, positioned to intercept AI agent actions before execution, can effectively secure AI agent deployments in production environments. This architecture is built upon three core principles: real-time operation, contextual awareness, and knowledge-based decision-making.

The specific key findings and contributions include:

1. Pre-invocation Interception as a Critical Control Point: The research demonstrates that placing a security guardrail at the exact moment an AI agent intends to call a tool or perform an action is highly effective. This "firewall for agents" prevents malicious or unintended behaviors from materializing, offering a proactive defense rather than a reactive response.
2. Efficacy of Specialized Small Language Model (SLM) Ensembles: The talk validates the concept of using an ensemble of multiple, specialized SLMs as "security judges." This approach significantly outperforms a single large language model (LLM) in terms of speed, accuracy, and cost-efficiency. Each SLM can be fine-tuned for a specific security concern (e.g., instruction violation, data security, threat detection), leading to more precise and robust classifications.
3. Significant Accuracy Gains through LoRA Fine-tuning: Practical results presented during the talk showcased a dramatic improvement in detection accuracy for instruction violation. For instance, a commonly used SLM like Mistral, which initially showed an F1 score around 85%, achieved "almost 100%" accuracy after targeted LoRA (Low-Rank Adaptation) fine-tuning. This highlights the effectiveness of specialized training for specific security tasks.
4. The Power of a Self-Improving, RAG-Based Memory System: The integration of a Retrieval Augmented Generation (RAG) system that learns an organization's unique patterns of tool invocation proves instrumental. By semantically searching past approved and rejected actions, the system provides crucial context, drastically reduces false positives, and adapts its decision-making over time, making the security posture more tailored and less disruptive.
5. A Scalable Microservices Architecture for AI Security: The proposed architecture is inherently modular, treating each SLM judge as an independent microservice. This design ensures that the system can scale efficiently, accommodate new risks, and be updated or modified without impacting other components, addressing the dynamic nature of AI threats.

Collectively, these findings present a robust and practical framework for securing AI agents, moving beyond theoretical concerns to offer a deployable solution that addresses the unique challenges of AI agent autonomy.

Technical Deep Dive

▶ Watch: Deep dive into Agent Goal Hijack and Rogue Agent risks. (5:45)

The proposed security architecture is designed as a runtime gate that intercepts an AI agent's intent before it translates into an action. This "firewall for agents" is a mandatory component, baked into the infrastructure, ensuring every tool invocation passes through it. The speaker noted that many AI vendors, such as Claude and Cursor, are already implementing similar "hooks" or junctions where such interception can occur.

The core of the system comprises three main components:

1. The Interception Layer: This layer sits between the Large Language Model (LLM) reasoning process (the "brain" of the agent) and the actual tool execution (the "hands"). When an agent decides to use a specific tool or make an API call, this intent is captured, along with the agent's reasoning process and relevant contextual data. The interception is designed to be real-time (very fast), contextual (understanding who is performing the action, what agent type, what data is involved), and knowledge-based (drawing on specialized security understanding).

1. The Ensemble of Small Language Model (SLM) Security Judges: Instead of relying on a single large, general-purpose LLM, the architecture employs multiple, fine-tuned Small Language Models (SLMs), referred to as "security judges." The rationale for using SLMs (typically 4 to 12 billion parameters) over larger models like Claude Opus 4.6 is threefold:

• Speed: SLMs are significantly faster, crucial for maintaining low latency in a runtime security solution.
• Specialization: Just as one wouldn't go to a general doctor for heart surgery, a specialized model is more effective at identifying specific security patterns. Each judge is trained for a particular type of risk.
• Cost: Running multiple SLMs is considerably more cost-effective than deploying and operating a single, large LLM.

The three primary security judges identified are:

• Instruction Violation Judge: This SLM is tasked with identifying instances where the agent deviates from its intended script or purpose. It covers risks like ASI 10 (Rogue Agent). An example cited was an agent instructed to summarize documents, but then attempting to send an email to a third party.
• Data Security Judge: This judge focuses on detecting the leakage of sensitive information, including Personally Identifiable Information (PII), credentials, or other confidential data. It's trained on specific patterns associated with such data.
• Threat Detection Judge: This SLM is designed to identify adversarial footprints within the agent's session, such as prompt injection attempts or jailbreak techniques.

These SLMs operate in parallel to save time and provide a multi-layered defense. Each judge is trained on different datasets, decision sets, and prompts, operating independently to ensure comprehensive coverage.

The training process for these SLMs is rigorous:

• Data Collection: A combination of publicly available research data (OWASP, MITRE), security patterns, and synthetic data (generated using publicly available libraries and known techniques) is used.
• Labeling: To ensure high-quality training data, three "high-thinking" large LLMs are employed to label examples. If all three agree on a label, it's marked with high certainty; otherwise, certainty is lowered.
• Fine-tuning: The chosen SLMs (e.g., Mistral, Llama 2) are then fine-tuned using LoRA (Low-Rank Adaptation). LoRA is a technique that efficiently adapts pre-trained models to new tasks by injecting trainable low-rank matrices into the transformer architecture, reducing the number of trainable parameters and computational cost. Confidence scores are also calibrated during this stage.
• Quality Validation: The final stage involves calculating standard model performance metrics like accuracy, precision, and recall. Crucially, a human expert also reviews the data to provide a sanity check and ensure the model behaves as expected.

1. The Memory System (Retrieval Augmented Generation - RAG): This component is critical for reducing false positives and adapting the security decisions to the specific context and behavior of an organization.

• Every tool invocation decision (approved or rejected) is stored in a repository, gradually building the system's understanding of the organization's operational norms.
• When a new request comes in, a semantic search is performed to find the most similar past invocations. If a similar behavior was approved 50 times in the past, it provides a strong signal for approval again.
• The system also incorporates a feedback loop: in cases requiring a human-in-the-loop (HITL) decision, the system learns from this manual intervention, further refining its knowledge base. This adaptive learning allows the system to "fit ourselves into the organization," as stated by Kaduri.

The Decision Flow:

The intercepted call (tool, intent, agent, and organizational context) is simultaneously evaluated by the parallel SLM judges. Their classifications, along with their certainty scores, are then passed to a thresholding mechanism.

• Auto-Deny: Actions with a very high certainty score (e.g., >95%) of being malicious or unintended are automatically blocked.
• Auto-Approve: Actions with a very low certainty score (e.g., <50%) of being malicious are automatically approved.
• RAG for In-between Cases: For decisions falling between these thresholds, the RAG memory system is consulted. If a strong, similar historical precedent exists for approval or denial within the organization's specific context, that decision is applied.
• Human-in-the-Loop (HITL) Escalation: If the RAG system doesn't provide a clear signal or if the certainty is ambiguous, the request can be escalated to a human expert for review. The goal is to keep HITL interventions as low as possible.

The RAG system is continuously updated with every decision, further enriching its understanding of organizational behavior.

Scalability and Innovation:

The architecture is designed as a microservices system, allowing individual SLM judges to be scaled, updated, or replaced independently. This modularity ensures fast development cycles, lower maintenance, and efficient integration of new risks or patterns. Kaduri highlighted three key innovative aspects: pre-invocation interception, the specialized SLM ensemble, and the self-improving RAG system. While each component might not be entirely novel in isolation, their combination forms a powerful and highly effective runtime security application for AI agents.

Demo / Proof of Concept

▶ Watch: Introducing the real-time, contextual security gate for agents. (7:45)

While the talk did not feature a live, interactive demonstration of the system in action, Bar Kaduri presented compelling data that served as a proof of concept for the effectiveness of their fine-tuning methodology, specifically for the instruction violation judge.

Kaduri displayed a chart showing the performance of several common small language models before specialized training. For the task of identifying instruction violations, these models exhibited decent but not production-ready F1 scores, with the best performer, Mistral, achieving an F1 score in the range of approximately 85%. The running durations for these models were also presented, indicating their inherent speed was suitable for runtime application.

The critical part of the demonstration involved showing the impact of their LoRA (Low-Rank Adaptation) fine-tuning process. After applying LoRA to the Mistral model, its F1 score for detecting instruction violations dramatically improved to "almost 100%." This significant boost in accuracy, from a baseline of ~85% to near-perfect detection, underscored the power of their specialized training approach. Kaduri emphasized that even a 10% chance of blocking a wrong action in production is unacceptable for users, making such accuracy improvements essential.

This data-driven demonstration effectively validated the hypothesis that fine-tuning SLMs with targeted data and techniques can yield highly accurate and reliable security judges, especially when combined with the contextual awareness of the RAG system, to achieve robust runtime security for AI agents.

Defensive Implications

▶ Watch: Architectural overview: SLM security judges intercepting agent intent. (9:45)

The security architecture presented by Bar Kaduri offers several critical implications for organizations seeking to safely deploy and manage AI agents:

• Proactive Runtime Security: The most significant implication is the shift from reactive to proactive security. By intercepting agent intents before tool execution, organizations can prevent malicious or unintended actions from ever occurring. This "pre-invocation interception" acts as a crucial last line of defense, mitigating risks like Agent Goal Hijack (ASI 1) and Rogue Agent (ASI 10) in real-time.
• Context-Aware Guardrails: The emphasis on contextual understanding is paramount. Security decisions are not binary; they depend on the agent's role, the user's identity, and the data involved. Defenders can implement fine-grained policies (e.g., a developer viewing customer data might be anomalous, but a customer success engineer doing so is acceptable) that adapt to specific operational contexts, reducing false positives and operational friction.
• Specialized and Efficient Threat Detection: The use of an ensemble of specialized Small Language Models (SLMs) allows for highly accurate detection of diverse threats. Instead of a single, less precise general model, defenders benefit from judges specifically trained for instruction violations, data security, and adversarial attacks (like prompt injection). This specialization leads to more effective threat identification without the computational overhead of large LLMs.
• Adaptive Security Posture: The Retrieval Augmented Generation (RAG) memory system provides a powerful mechanism for adaptive security. It learns an organization's unique patterns of tool usage and agent behavior over time. This continuous learning reduces the need for constant manual rule adjustments and significantly lowers false positive rates, allowing security teams to focus on genuine threats and operational anomalies.
• Enablement for AI Agent Deployment: Ultimately, this architecture empowers organizations to deploy AI agents in production environments with greater confidence. By providing a robust, intelligent security layer, it addresses the core anxieties around agent autonomy and unpredictability, transforming agents from potential liabilities into secure, productive assets. The goal is to allow users to "sleep at night" knowing their agents are operating within defined, secure boundaries.
• Scalability and Future-Proofing: The microservices-based design ensures that the security solution itself is scalable and adaptable to evolving threats. As new AI risks emerge, new specialized SLMs can be developed and integrated without disrupting the entire system, offering a future-proof approach to AI agent security.

Key Takeaways

• AI Agents Introduce Novel & Significant Attack Surfaces: The rise of autonomous AI agents creates unique security challenges, notably Agent Goal Hijack (ASI 1) and Rogue Agent (ASI 10), which demand new, AI-native security solutions beyond traditional application security.
• Pre-Invocation Interception is Critical for Runtime Security: Implementing a security gate that intercepts an AI agent's intent to use a tool before execution is crucial for preventing malicious or unintended actions in real-time.
• Specialized SLM Ensembles Offer Superior Performance: Utilizing an ensemble of fine-tuned Small Language Models (SLMs), each specialized for a specific security risk (e.g., instruction violation, data security, threat detection), provides faster, more accurate, and cost-effective protection than a single general-purpose large language model.
• Adaptive RAG Systems Enhance Context and Reduce False Positives: A self-improving Retrieval Augmented Generation (RAG) memory system, which learns an organization's historical agent behaviors, is essential for providing contextual decision-making and significantly reducing false positives over time.
• LoRA Fine-tuning Dramatically Improves SLM Accuracy: Targeted fine-tuning techniques like LoRA (Low-Rank Adaptation) can drastically boost the accuracy of SLMs for specific security tasks, with demonstrated improvements from ~85% to nearly 100% F1 scores for instruction violation detection.
• A Multi-Layered, Adaptive Architecture Enables Secure AI Agent Deployment: The combination of pre-invocation interception, specialized SLM judges, and a self-improving RAG system forms a robust, scalable, and adaptive security framework that allows organizations to confidently deploy AI agents in production.

About the Speaker(s)

Bar Kaduri is a seasoned cybersecurity professional with 14 years of experience in the field. She began her career as a threat intelligence analyst before spending five years as a malware analyst at Check Point. Later, she joined Orca Security, where she served as a cloud security analyst and ultimately became the Head of Research and Detection, during which time her team identified over 25 vulnerabilities in cloud and companion products. Recognizing the paradigm shift required for AI security, Bar transitioned to Capsule Security, where her work focuses on developing AI-driven solutions for securing AI agents at runtime. Her expertise spans deep technical analysis, vulnerability research, and the strategic implementation of cutting-edge security technologies.

Lidan Hazout was acknowledged by Bar Kaduri as the "brain and the wind and the spirit behind a lot of the innovation" presented in the talk. While not present at the conference, his contributions were integral to the development of the ensemble of fine-tuned small language models and the overall security architecture discussed.