Context-Aware Authorization for Agentic Tool Calls (Agent Memory Informed Authorization)

Context-Aware Authorization for Agentic Tool Calls (Agent Memory Informed Authorization)

Speakers: Robert, C1.AI

Conference: fwd:cloudsec

YouTube: https://www.youtube.com/watch?v=YOK5gJVateg

Overview

In an era where artificial intelligence agents are increasingly integrated into daily workflows, both assisting human employees and operating autonomously, the challenge of securing their access to organizational resources becomes paramount. Robert from C1.AI addresses this critical issue in his fwd:cloudsec talk, "Context-Aware Authorization for Agentic Tool Calls," which he also refers to as "Agent Memory Informed Authorization." The core premise is to move beyond static, broad permissions for AI agents and instead leverage their dynamic "memory" to inform real-time, context-aware authorization decisions.

The presentation takes a builder's perspective, focusing on how organizations can safely deploy and manage an "agentic workforce." C1.AI, an IGM governance product company, is on a mission to secure this emerging landscape, recognizing that agents pose a significant, perhaps even greater, risk than human identities from an identity and access management (IAM) standpoint. The talk delves into the inherent difficulties of managing agent permissions at scale and proposes a novel, technically robust architecture that combines advanced AI capabilities with traditional security principles and a specialized logic programming language.

This approach is crucial because traditional IAM systems, designed for human users with relatively static roles and entitlements, are ill-equipped to handle the dynamic, high-volume, and often ambiguous nature of agent interactions. By dynamically tailoring an agent's permissions based on its current operational context – derived from its internal memory – the system aims to drastically reduce the "blast radius" of potential security incidents while simultaneously empowering agents to perform their tasks efficiently. This talk offers a pragmatic pathway for organizations to embrace agent technology without compromising their security posture.

Background

▶ Watch: Welcome and Introduction to Agent Memory Informed Authorization (0:00)

The proliferation of AI agents in enterprise environments introduces a fundamental security dilemma: how to grant agents sufficient access to perform their tasks without exposing an organization to undue risk. Robert highlights the inadequacy of current approaches and common recommendations. For instance, NIST's optimistic suggestions for maintaining "full visibility" and "cataloging permissions" for agents are critiqued as "unrealistic," "inadequate, and hopeless." The sheer volume and dynamic nature of agent permissions make a manual or static cataloging approach unfeasible. An agent's access needs can change moment-to-moment based on its current task, making traditional raw entitlements—broad, static permissions—both inefficient and insecure.

The problem space is twofold: first, the sheer number of potential permissions an agent might need is astronomical, making a simple whitelist or database of entitlements unmanageable. Second, agents require dynamic access. Organizations want to empower agents to access resources autonomously, but only precisely when and for the specific purpose required, not with blanket, carte blanche access. This challenge is exacerbated by the fact that many existing IAM systems, while perhaps adequate for human users, are not designed for the higher scale and volume of interactions inherent to an agentic workforce. The core issue, Robert emphasizes, is granularity: how to provide fine-grained, context-specific permissions rather than overly broad ones.

Prior attempts at "intent-based authorization" often fall short due to their reliance on real-time Large Language Model (LLM) inference in the authorization path, which is not scalable. For example, tools like Crab Trap, an LLM-in-the-loop proxy, might work for low-volume scenarios or those with generous token budgets, but they fail to meet the reliability and scalability demands of large organizations. Such approaches also lack crucial features like versionability and auditability of policy decisions, making it difficult to understand why an LLM made a particular authorization choice at a specific point in time. The need, therefore, is for a system that can dynamically author policies based on contextual input, but then statically evaluate them at runtime for speed, reliability, and auditability.

Key Findings

▶ Watch: Why NIST's agent authorization recommendations are unrealistic (2:00)

The central discovery and contribution of Robert's talk is the concept of Agent Memory Informed Authorization, which dynamically generates and refines authorization policies using an agent's internal memory as a contextual input. This approach addresses the inherent challenges of securing AI agents by providing granular, dynamic permissions without sacrificing performance or auditability.

1. Agent Memory as Contextual Input: The talk identifies agent memory—which can encompass conditional, time-ranged, or predicate-exposed facts—as a rich source of runtime context. This memory is leveraged to dynamically author authorization policies that reflect the agent's current task or intent. The key insight is that memory can be instrumented, much like a security control, to understand an agent's operational focus.

1. LLMs for Offline Policy Authoring and Semantic Understanding: Large Language Models (LLMs) play a crucial role, not in the real-time authorization loop, but in the asynchronous policy generation phase. LLMs excel at tasks like vector clustering to understand the semantic relationships between an agent's stated intent (e.g., "user management") and specific resources (e.g., "employees" database). This capability allows for the creation of precise, intentional policies that account for the ambiguities of vocabulary across different systems.

1. Dynamic Policies, Static Evaluation (Read-Split vs. Write-Split): A core architectural principle is the separation of policy authoring (write path) from policy evaluation (read path). Policies are dynamically generated (often asynchronously and with LLM assistance) but are then evaluated statically and synchronously at runtime. This ensures that the performance-critical authorization decision is fast and deterministic, avoiding the latency and non-determinism of real-time LLM inference.

1. Datalog for Provable and Performant Policy Enforcement: Robert advocates for Datalog as the policy language. Datalog, a logic programming language, is "barely a programming language" in its simplicity, making it decidable and terminating. This property is crucial for security, as it prevents unbounded recursion or infinite loops that could be introduced by more complex languages. Furthermore, Datalog's suitability for graph traversals and its ability to aggregate facts and policies into a single program enable proofs about policy behavior (e.g., a policy will only narrow permissions, never expand them). LLMs are also shown to be exceptionally good at generating syntactically and semantically correct Datalog.

1. Narrowing Principle for Security: Crucially, the system design ensures that agent memory and dynamically generated policies only narrow existing entitlements. They never expand an agent's baseline access. This principle provides a strong security guarantee, ensuring that even if memory is compromised or misinterpreted (e.g., via prompt injection), the agent cannot gain access to resources it wasn't already broadly entitled to, thereby maintaining a postural improvement over traditional broad access.

Technical Deep Dive

▶ Watch: Introducing agent memory for contextual authorization policies (3:40)

The fundamental problem addressed is the inability of traditional Identity and Access Management (IAM) systems to securely and efficiently manage permissions for AI agents. These systems struggle with the sheer volume of potential agent interactions, the dynamic nature of an agent's operational context, and the need for granular access control. Robert introduces Agent Memory Informed Authorization as a solution, built on a robust architectural split between policy generation and enforcement.

At the heart of the proposed system is the concept of using an agent's memory as a dynamic input for authorization. Unlike strict factual logs, agent memories can be complex, incorporating conditional logic, time-based validity, or exposed predicates. These predicates—which might describe the agent's current task, the data it's processing, or the user it's serving—become the contextual cues for authorization logic. For example, an agent's memory might indicate it's currently focused on "user management," which can then be semantically linked to resources like "employees" databases.

The system's architecture is divided into two primary paths: the write path (policy authoring) and the read path (policy evaluation). This separation is critical for scalability and performance.

Write Path (Asynchronous Policy Authoring):

In the write path, various data sources (which could include the agent's internal memory logs, user inputs, or external system states) feed into an LLM. This LLM's role is not to make real-time authorization decisions, but to extract structured facts and perform semantic vector clustering. For instance, it can understand that an agent's task of "user management" semantically relates to the "employees" database. Crucially, the LLM then uses this understanding to author authorization policies. These policies are not arbitrary code but are generated in a highly constrained, deterministic language, which Robert strongly advocates to be Datalog. The resulting Datalog policies are then written to an authorization system, which updates the enforcement points. This process is asynchronous, allowing for complex LLM operations without impacting runtime performance.

Read Path (Synchronous Policy Evaluation):

The read path is designed for speed and reliability. When an agent attempts a tool call or makes an API request, this request is intercepted by an egress proxy. This proxy is the enforcement point, and it performs several key functions:

1. Introspection: It inspects the outbound request, analyzing its contents and target resources.
2. Semantic Vector Search: It can perform quick semantic lookups over the request content, similar to the LLM in the write path, but using pre-computed embeddings.
3. Policy Evaluation: The proxy evaluates the request against the latest snapshot of dynamically authored policies (represented in Datalog). This evaluation is static and deterministic, leading to an "allow" or "deny" decision.

A critical design constraint is that agent memory and the resulting policies are used only to narrow existing entitlements, never to expand them. If an agent already has baseline access to a broad set of resources, its current context (from memory) might narrow that access to only the specific subset relevant to its current task. This principle acts as a strong security guardrail, limiting the potential blast radius of a compromised agent or a policy misinterpretation.

Datalog as the Policy Language:

Robert dedicates significant attention to justifying Datalog for policy definition. Its advantages are compelling:

• Decidable and Terminating: Unlike general-purpose programming languages (e.g., TypeScript), Datalog inherently guarantees that any program will terminate and produce a result. This is paramount for a fast, reliable authorization proxy, preventing infinite loops or unbounded recursion—issues that could be exploited or simply degrade performance.
• Proof Capabilities: The simplicity and decidability of Datalog allow for formal proofs about policy behavior, such as proving that a policy will only narrow permissions.
• Graph Traversal: Authorization often involves complex relationships, such as group hierarchies ("this group contains this group which has this role") or management chains. Datalog is exceptionally well-suited for modeling and traversing these recursive graphs safely, without the risks associated with recursive functions in other languages.
• LLM Affinity: LLMs are "better than people" at generating syntactically and semantically correct Datalog due to its static corpus and lack of statistical variation. This makes the LLM-driven policy authoring process highly reliable.
• Aggregatable Facts and Policies: Both runtime facts (e.g., "within the last hour this agent has been operating on our customer database") and the policies themselves can be expressed as Datalog tuples. This allows them to be concatenated into a single Datalog program, providing a deterministic "computer program that will tell you what the authorization decision was at that point in time." This is invaluable for auditability.

System Workflow:

1. A user interacts with an agent, typically via natural language.
2. The agent identifies APIs or Managed Cloud Provider Services (MCPs) to call.
3. The agent's natural language input and subsequent actions update its memories. These memories, along with other contextual inputs, are asynchronously observed by a component (referred to as a "Z agent" in C1.AI's context).
4. The Z agent, potentially with LLM assistance, asynchronously rewrites and updates policies (in Datalog) based on these memories.
5. These updated policies are pushed to the Zoxroxy (egress proxy), which maintains a snapshot of current authorization rules.
6. When the agent makes a tool call or API request, it hits the Zoxroxy.
7. The Zoxroxy evaluates the request against its Datalog policy snapshot, making a fast, deterministic allow/deny decision.
8. In "iffy" cases (the "gray area" between allow and deny), the system can programmatically generate an Access Request (AR) for a human or even another agent manager, providing a mechanism for human-in-the-loop oversight.

This system design streamlines workflows by eliminating the need for users or agents to explicitly request access (e.g., "I need access to our dev database for credit card fraud detection"). Instead, the agent's context implicitly informs and refines its permissions.

Demo / Proof of Concept

▶ Watch: LLMs for policy authoring, not runtime evaluation (4:30)

While Robert's talk provides a detailed conceptual architecture and technical justification for Agent Memory Informed Authorization, it does not feature a live, functional demonstration of the system in action. The speaker explicitly mentions a diagram illustrating the "request layout" was rendered using a "shader that I made in metal," but clarifies this was for visual representation rather than a functional proof of concept of the full authorization pipeline.

However, the talk extensively describes the intended functionality and flow. For instance, the speaker illustrates with a code example of Datalog on screen, demonstrating how facts and rules can be expressed in this logic programming language. The "gray area" concept, where an "iffy" request could trigger an Access Request (AR) to an agent's manager (human or agent), highlights a potential interactive element of the system, even if not explicitly demoed. The overall system design diagram, despite a minor rendering quirk, visually conveys the asynchronous policy updates and synchronous proxy-based enforcement, providing a clear conceptual proof of how the components would interact.

Defensive Implications

▶ Watch: Limitations of LLM-in-loop egress proxies like Crab Trap (5:15)

The Agent Memory Informed Authorization framework presents several significant defensive implications for organizations grappling with the security challenges of an agentic workforce:

1. Reduced Blast Radius: By dynamically narrowing an agent's permissions based on its current context and memory, the system drastically reduces the "blast radius" of a compromised agent. Robert provides the example of a marketing person using Managed Cloud Provider Services (MCPs): previously, their agent might have had broad access via their personal credentials, making it a high-value target for supply chain attacks. With this system, the agent's access is constrained to only what's necessary for its immediate task, even if the underlying entitlements are broader. This limits the damage an attacker could inflict.

1. Mitigation of Credential Exposure: A crucial improvement is the elimination of agents needing to store or manage their own credentials. Instead, the egress proxy injects the necessary credentials into the request at runtime. This directly addresses vulnerabilities like those seen in npm hacks, where compromised developer machines or packages could exfiltrate tokens or credentials stored on the file system. By centralizing credential management and injection, the system hardens the agent's operational environment.

1. Enhanced Postural Security Against Prompt Injection/Memory Poisoning: While acknowledging that no system is perfect, Robert highlights that the "narrowing principle"—where agent memory only refines existing permissions and never expands them—provides a significant security improvement against prompt injection or memory poisoning. Even if an attacker successfully manipulates an agent's memory, the worst-case scenario is that the agent loses access or operates within its already defined (and ideally minimal) baseline, rather than gaining new, unauthorized capabilities. This provides a more secure posture compared to systems where untrusted input could directly lead to privilege escalation.

1. Deterministic and Auditable Authorization Decisions: The use of Datalog for policies, combined with the clear separation of the write and read paths, ensures that authorization decisions are deterministic and auditable. Datalog's decidability guarantees that policies will terminate, preventing unpredictable behavior. Furthermore, the ability to aggregate facts and policies into a single Datalog program means that at any point in time, the exact authorization decision and the rationale behind it can be reconstructed and proven. This is invaluable for forensic analysis, compliance, and understanding "why" a particular decision was made, especially when compared to opaque LLM-in-the-loop systems.

1. Static Validation and Real-time Feedback: Robert notes that using simple, syntactical frameworks like Datalog allows for extensive static validation. Buggy code, hallucinated verbs, or incorrect nouns can be checked against defined schemas and ontologies. The integration of Language Server Protocol (LSP)-like capabilities means agents can receive real-time feedback as they generate or modify policies, preventing invalid policies from ever being uploaded to the enforcement system. This proactive validation significantly reduces the risk of policy errors leading to security gaps.

1. Streamlined and Secure Workflows: By making authorization context-aware, the system streamlines internal workflows. Agents can dynamically adapt their access based on their current task, reducing the friction of manual access requests. This empowers the agentic workforce to operate more efficiently while maintaining a strong security perimeter, striking a balance between productivity and protection.

Key Takeaways

• Agent memory is a powerful contextual input for dynamic authorization: Leveraging an agent's internal state allows for real-time, context-aware policy decisions that go beyond static entitlements.
• Datalog is an ideal policy language for agent authorization: Its decidable, terminating nature, suitability for recursive graph traversals, and LLM compatibility make it highly secure, performant, and auditable for complex authorization logic.
• LLMs should be used for offline policy authoring, not runtime enforcement: Large Language Models are excellent for semantic understanding and generating precise policies asynchronously, but the runtime authorization path requires fast, deterministic, static evaluation by a dedicated proxy.
• The "narrowing principle" is crucial for security: Agent memory and context-aware policies should only refine and restrict existing permissions, never expand them, providing a strong security guardrail against misuse or compromise.
• Separation of concerns (write/read path) is key for scalability and reliability: Asynchronous policy generation allows for complex computations, while synchronous, static evaluation at an egress proxy ensures fast and predictable authorization decisions.
• This approach significantly enhances the security posture of an "agentic workforce": By reducing blast radius, eliminating hardcoded credentials, and providing auditable, context-aware access, organizations can safely embrace AI agents.

About the Speaker(s)

Robert, representing C1.AI, is a key figure in the company's mission to secure the burgeoning "agentic workforce." C1.AI, recently rebranded from C1, specializes in IGM (Identity Governance and Management) governance products. Robert approaches the challenges of agent security from a "builder's point of view," focusing on practical solutions for organizations to safely integrate and manage AI agents. His work at C1.AI involves developing security products that not only serve external clients but also address the company's internal needs for securing its own agent usage. His expertise lies in translating complex security problems posed by AI agents into actionable architectural and policy frameworks.