Agents Unleashed - Can We Control What We've Created?

Agents Unleashed - Can We Control What We've Created?

Speakers: Trina Ford, Chief Information Security Officer, iHeartMedia; Niel Harper, Chief Trust Officer, Hugo; Priya Mouli, Chief Information Security Officer, Sheridan College (Moderator)

Conference: Black Hat MEA

YouTube: https://www.youtube.com/watch?v=Z2lxjSANrt8

Overview

The rapid proliferation of autonomous and AI-driven agents within the enterprise presents a dual challenge: maximizing their utility while simultaneously ensuring their safety and maintaining organizational control. This panel discussion, "Agents Unleashed - Can We Control What We've Created?", delves into the critical security implications of deploying these intelligent systems. Moderated by Priya Mouli, CISO of Sheridan College, with insights from Trina Ford, CISO of iHeartMedia, and Niel Harper, Chief Trust Officer at Hugo, the session provides a timely exploration of how organizations can unlock the substantial value offered by AI agents without inadvertently ceding control or escalating risk.

The core premise of the talk revolves around the stark reality that while agents move fast, the associated risks often move even faster. As AI agents assume more tasks, from customer service escalations to critical system modifications, the imperative to establish robust safeguards becomes paramount. The discussion explores practical controls, governance frameworks, and strategic considerations essential for balancing innovation with rigorous oversight in an evolving landscape where intelligent agents are increasingly integral to business operations.

The panelists underscore that while AI agents represent a new frontier, many of the fundamental security principles that apply to human users and traditional IT systems remain relevant, albeit requiring adaptation. They emphasize the need for proactive measures, drawing parallels to established practices in identity management, incident response, and third-party risk. The conversation serves as a crucial guide for security professionals grappling with the complexities of managing autonomous entities within their organizational ecosystems.

Background

▶ Watch: Scenario: Overly helpful AI escalates wrong issue (2:45)

The advent of AI agents marks a significant shift in enterprise operations, promising unprecedented efficiencies, cost savings, and the ability to perform complex tasks with high speed and scale. These agents, designed to act autonomously, are increasingly being deployed across various functions, from automating customer interactions to managing backend infrastructure. However, this rapid adoption often outpaces the development and implementation of adequate security and governance frameworks, creating a fertile ground for unforeseen risks.

The core problem, as highlighted by the panel, is the inherent tension between the desire for useful autonomy and the necessity of maintaining control. A vivid scenario introduced by the moderator illustrates this dilemma: an AI agent, in its attempt to be "overly helpful," escalates the wrong customer issue to the board, creating significant reputational damage without any malicious intent, breach, or data loss. This hypothetical, yet plausible, situation underscores the unique challenge posed by AI agents: their actions, even when well-intentioned, can have profound organizational impacts if not properly constrained and governed.

Historically, organizations have grappled with similar challenges with new technologies, such as the adoption of cloud computing or the management of third-party vendors. In each case, the initial rush for innovation often led to "shadow IT" or unmanaged risks. With AI agents, this risk is amplified by their autonomous nature, learning capabilities, and potential for rapid, unmediated actions. The panel references recent real-world incidents, such as the Anthropic attack – a cyber espionage campaign that reportedly leveraged its own Claude AI model to autonomously execute 80-90% of the attack – as a stark reminder of the sophisticated threats that can arise from uncontrolled AI capabilities. This context sets the stage for a discussion on critical safeguards, including least privilege, observability, reversible changes, and governance that survives production, all aimed at ensuring that AI agents remain aligned with security objectives.

Key Findings

▶ Watch: Balancing useful AI autonomy with unacceptable business risk (5:20)

The panel discussion yielded several key findings and recommendations for controlling autonomous AI agents, emphasizing a proactive and integrated security approach.

First, the panelists stressed the non-negotiable need for foundational technical controls before agent deployment. Niel Harper introduced two critical early-stage controls: output gates and rate limitations. Output gates act as a security-controlled service layer that mediates every final action of an autonomous agent, comparing it against established enterprise policies. This prevents agents from performing restricted actions, such as initiating financial transactions above a certain threshold (e.g., $500) or deleting critical production systems. Rate limitations, on the other hand, are designed to prevent issues like hallucination escalation or denial of service by throttling or stopping agents that exhibit anomalous or high-volume harmful actions.

Second, the discussion highlighted the crucial balance between utility and risk. Trina Ford articulated that the line between useful autonomy and unacceptable risk is drawn by assessing an agent's value proposition – its contribution to cost savings, efficiency, and streamlining operations – against its potential business disruption. Useful agents perform tasks that free up human resources, such as creating tickets, triaging alerts, or performing analytical tasks. Unacceptable risk arises when an agent's actions have a high impact on the business, necessitating robust administrative controls like policies, standards, and guidelines, alongside a clear understanding of risk acceptance.

Third, a recurring and powerful theme was the imperative to treat AI agents much like human identities, albeit with even greater scrutiny. Both Harper and Ford underscored that AI is no different from other new technologies in requiring established security practices. They argued that agents must have an identity, be subject to least privilege, authentication, and authorization, and have their actions logged and monitored. This approach acknowledges that while agents are non-human, they can perform autonomous tasks, reason, and adapt, making their management akin to, or even more critical than, managing human users due to their potential for rapid, widespread impact.

Finally, the discussion emphasized the critical role of governance and incident response in the AI agent lifecycle. Neil Harper advocated for every proposed agent to be registered with a central governance function (e.g., an IT steering committee or AI board) and risk-rated. This rating dictates the subsequent security measures, including architecture reviews and penetration tests. Furthermore, organizations must integrate AI agent incidents into their existing incident response frameworks, updating playbooks, plans, and forensic analysis capabilities to address agent-specific behaviors and potential regulatory reporting thresholds. The panel collectively reinforced that a "human in the loop" approach, emphasizing human discretion and the ability to reverse agent actions, is paramount for responsible AI deployment.

Technical Deep Dive

▶ Watch: Non-negotiable governance for AI agents, incident response (10:00)

The technical controls and architectural considerations discussed by the panelists offer a framework for securing autonomous AI agents. The core philosophy centers on embedding security at every stage of the agent's lifecycle, from design to deployment and ongoing operation.

1. Output Gates and Policy Enforcement:

Niel Harper's concept of output gates represents a critical technical control. This involves creating a security-controlled service layer that acts as an intermediary for every final action an autonomous agent intends to take. Before any action is executed, the agent's proposed output is compared against a predefined enterprise policy. This policy is granular and context-aware, designed to restrict specific types of actions or limit their scope. For example:

• Financial Transactions: If an agent is authorized to initiate financial transactions, the policy might restrict the transaction amount to "less than $500." Any attempt by the agent to exceed this limit would be blocked by the output gate.
• Production System Modifications: For agents interacting with production systems, the policy could prevent actions like "deleting or restricting a production system which is tagged as critical." This requires accurate tagging and classification of assets within the enterprise.

Implementing this requires a robust policy engine that can parse agent requests, understand their context, and evaluate them against a dynamic rule set. This service layer would typically sit between the agent's execution environment and the target system or API.

2. Rate Limitations for Stability and Security:

To prevent unintended consequences such as hallucination escalation, denial of service (DoS), or rapid, unchecked harmful actions, rate limitations are essential. This control monitors the volume and frequency of an agent's actions. If an agent begins to exhibit an unusually high volume of requests or actions that deviate from its expected operational profile, the rate limiter can:

• Throttle: Reduce the speed at which the agent can perform actions.
• Stop: Temporarily or permanently halt the agent's operations.

This mechanism requires behavioral analytics and anomaly detection capabilities, often integrated into the agent's monitoring framework. Baselines of normal agent behavior must be established to effectively identify deviations that warrant intervention.

3. Agent Identity and Least Privilege:

A foundational security principle, least privilege, is particularly challenging yet vital for AI agents. The panelists stressed that agents, despite being non-human, must possess a distinct identity. This identity is then used to enforce specific authentication and authorization rules.

• Specific Access: An agent should only be granted the minimum necessary access required for its designated task. If an agent's role is to analyze log files, it should not have permissions to modify production databases.
• Time-Limited Access: Access can be further constrained by time, similar to temporary credentials for human users or service accounts. This reduces the window of opportunity for misuse if an agent's identity is compromised or if it veers off its intended path.
• System-Level Actions: Trina Ford emphasized limiting agents not just to specific systems but to specific actions within those systems. This requires granular access control lists (ACLs) or Role-Based Access Control (RBAC) applied at the API or service level, dictating what an agent can do (e.g., read, write, delete) rather than just where it can go.

This approach necessitates robust identity and access management (IAM) solutions capable of managing non-human identities, potentially integrating with existing enterprise IAM systems.

4. Reversible Actions and Blast Radius Containment:

A critical technical requirement for AI agents is ensuring that their actions are reversible. Niel Harper highlighted that agents should not be granted permissions that allow them to initiate irreversible actions. This is crucial for mitigating damage if an agent malfunctions or acts inappropriately.

• Logging and Monitoring: Comprehensive logging of all agent actions is paramount. This allows for forensic analysis to understand what an agent did and when, enabling human operators to trace its steps.
• Intervention Capabilities: Security teams must have the ability to intervene and reverse an agent's actions. This implies that critical operations performed by agents should have corresponding "undo" or rollback mechanisms.
• Limited Impact (Blast Radius): Trina Ford added that the impact of agent actions should be inherently small enough that even if an action cannot be immediately reversed, its blast radius is contained. This is achieved through careful architectural design and the application of least privilege. For example, an agent might operate within a sandbox or a highly segregated environment for sensitive tasks.

5. Governance Integration and Incident Response:

From a technical governance perspective, the panelists advocated for:

• Agent Registry and Risk Rating: Every agent proposed for deployment should be formally registered and undergo a risk rating process. This rating determines the rigor of subsequent technical reviews, including architecture reviews and penetration tests. High-risk agents (e.g., those that can modify production systems or initiate financial transactions) would require more extensive technical scrutiny.
• Integrated Incident Response: Existing incident response frameworks must be updated to specifically address AI agent-related incidents. This includes developing playbooks for agent misbehavior, defining procedures for forensic analysis of agent logs and behavior, and establishing thresholds for regulatory reporting if an agent causes a data breach or other reportable incident.
• Threat Modeling: Applying threat modeling methodologies to AI agents helps identify potential vulnerabilities and attack vectors specific to their design and operation, allowing for proactive control implementation. This includes considering risks like prompt injection, data poisoning, or model evasion attacks.

By integrating these technical controls and adapting existing security practices, organizations can build a more resilient and secure environment for deploying and managing autonomous AI agents, ensuring that innovation does not come at the cost of control.

Demo / Proof of Concept

▶ Watch: Implementing least privilege for agents: confinement, monitoring, reversible ... (15:20)

This session was presented as a panel discussion, featuring expert insights and strategic advice from the speakers. As such, there was no live demonstration or proof of concept presented during the talk. The focus was entirely on theoretical frameworks, practical advice, and strategic considerations for managing AI agents in an enterprise environment.

Defensive Implications

▶ Watch: Preparing for shadow AI: inventory, steering committee, risk ownership (19:00)

The insights shared by the panelists offer several critical defensive implications for organizations looking to secure their AI agent deployments:

1. Implement Robust Output Gates and Rate Limiters: Defenders must prioritize the implementation of technical controls like output gates and rate limitations. This means architecting a security-controlled service layer that mediates all critical agent actions, comparing them against granular enterprise policies. Simultaneously, deploy systems that can monitor agent behavior for anomalous activity and automatically throttle or halt agents exhibiting signs of hallucination escalation or potential denial of service attacks. This requires defining clear policy rules and establishing behavioral baselines for each agent.

1. Establish Comprehensive AI Agent Inventory and Governance: Organizations need to develop a formal process for registering, risk-rating, and approving every AI agent before deployment. This involves creating a central inventory of agents that includes details about their purpose, access levels, data handling, and risk classification. A dedicated governance function (e.g., an AI steering committee) should be responsible for overseeing this process, ensuring that architecture reviews and penetration tests are conducted commensurate with the agent's risk profile.

1. Integrate AI Agent Incidents into Incident Response: Security teams must update their existing incident response frameworks to specifically address AI agent-related incidents. This includes developing dedicated playbooks for agent misbehavior, data breaches caused by agents, or autonomous attacks. Forensic analysis capabilities need to be enhanced to collect and interpret agent-specific logs and behavioral data, and legal/compliance teams must understand the regulatory reporting thresholds for AI-induced incidents.

1. Enforce Strict Identity and Least Privilege for Agents: Treat AI agents as distinct, non-human identities within the organization's Identity and Access Management (IAM) system. Apply the principle of least privilege rigorously, granting agents only the specific access required for their designated tasks, for the minimum necessary duration. This means granular access controls at the system, application, and API levels, limiting not just what systems an agent can access, but what actions it can perform within those systems.

1. Prioritize Observability and Reversibility: Implement extensive logging and monitoring for all agent activities. This is crucial for understanding agent behavior, detecting anomalies, and enabling post-incident forensic analysis. Furthermore, design agent interactions such that critical actions are reversible by human operators. Agents should not be given permissions to perform irreversible operations without human oversight or a clearly defined rollback mechanism, thus containing the potential blast radius of any erroneous actions.

1. Proactively Address Shadow AI: Recognize that shadow AI is an emerging threat, akin to shadow IT. Defenders should implement strategies to discover unauthorized or unmanaged agents within the enterprise. This could involve network scanning, endpoint detection and response (EDR) solutions looking for unexpected AI process activity, and regular audits of cloud service usage to identify unsanctioned AI model deployments. Raising awareness among business units about the risks of deploying unmanaged agents is also critical.

1. Emphasize "Human in the Loop" and Judgment: While agents offer autonomy, the final layer of defense often remains human judgment. Design workflows that incorporate human oversight, particularly for high-impact or irreversible actions. This ensures that human discretion can override agent decisions when necessary, preventing scenarios like the "overly helpful AI" escalating the wrong issue to the board.

By adopting these defensive postures, organizations can proactively manage the risks associated with AI agents, turning their potential into a secure operational advantage rather than an uncontrolled liability.

Key Takeaways

• Treat AI Agents as Identities with Extreme Caution: AI agents, though non-human, must be managed with robust identity, authentication, authorization, and least privilege principles. Due to their autonomous and learning capabilities, they can pose even greater risks than human users if unchecked.
• Foundational Technical Controls are Non-Negotiable: Implement essential safeguards like output gates to mediate and restrict agent actions against enterprise policy, and rate limitations to prevent runaway processes, hallucination escalation, or denial of service.
• Comprehensive Governance and Risk Management are Paramount: Establish a formal process for registering, risk-rating, and approving all AI agents. Integrate agent-specific risks into existing incident response frameworks and threat modeling exercises, updating playbooks and forensic capabilities.
• Prioritize Observability and Reversibility of Actions: Ensure all agent actions are extensively logged and monitored, and design systems such that human operators can easily reverse agent decisions or contain their impact, limiting the blast radius of any erroneous behavior.
• Actively Combat Shadow AI: Proactively scan, inventory, and manage all AI agents within the enterprise to prevent the proliferation of unauthorized or unmanaged systems that introduce unquantified risks.
• Maintain "Human in the Loop" for Critical Decisions: Emphasize human oversight and discretion, especially for high-impact or irreversible actions, ensuring that technology serves human objectives without ceding ultimate control.

About the Speaker(s)

Priya Mouli served as the moderator for this insightful panel discussion. She is the Chief Information Security Officer (CISO) for Sheridan College, bringing her expertise in cybersecurity leadership to guide the conversation on managing risks associated with AI agents.

Trina Ford is the Chief Information Security Officer (CISO) of iHeartMedia. With a background that initially aimed for a CIO role, she found her calling in cybersecurity leadership. Her perspective emphasizes the balance between cost-effectiveness, efficiency, and risk acceptance when approving AI agent deployments, focusing on the value agents bring versus their potential for business disruption.

Niel Harper holds the position of Chief Trust Officer at Hugo. His experience includes being thrown into a massive ransomware attack on his very first day as a CISO, an experience he now looks back on as a "fun fact." He brings a practical, control-oriented approach to securing AI agents, advocating for robust technical and governance mechanisms from the outset.