Open Policy Agent (OPA) Intro & Deep Dive - Charlie Egan & Anders Eknert, Styra

Open Policy Agent (OPA) Intro & Deep Dive

Speakers: Charlie Egan, Developer Relations; Anders Eknert, Developer Relations; Styra

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=XtA-NKoJDaI

Overview

This talk, presented by Charlie Egan and Anders Eknert, both maintainers of the Open Policy Agent (OPA) project and members of the developer relations team at Styra, provides a comprehensive introduction and deep dive into OPA. It addresses the critical need for standardized, decoupled policy enforcement in modern, distributed systems. The speakers elaborate on OPA's core principles, its role as a policy engine, and its widespread adoption across various use cases, from Kubernetes admission control to application-level authorization and CI/CD pipelines.

The presentation aims to cater to both newcomers and existing OPA users, offering foundational concepts alongside updates on recent advancements, upcoming features, and significant performance improvements. It underscores OPA's value proposition in allowing organizations to express policies as code, enabling them to be version-controlled, collaboratively developed, and consistently applied across diverse technology stacks. This approach fosters greater agility, security, and auditability in managing complex authorization and governance requirements.

Ultimately, the talk highlights how OPA empowers developers and security teams to abstract policy logic away from application code, thereby standardizing enforcement, reducing development overhead, and enhancing overall system security. The discussion on OPA 1.0 and the impressive performance gains demonstrates the project's maturity and its ongoing commitment to providing a robust and efficient policy enforcement solution for the cloud-native era.

Background

▶ Watch: Defining policy as code (1:55)

The concept of "policy" in the context of this presentation extends far beyond traditional legal or human-readable guidelines. Here, policy is understood as a set of definitive rules that govern behavior and access within technical systems. This encompasses everything from authorizing user actions and granting tenant access in Kubernetes to defining custom rules in CI/CD jobs and implementing intricate business logic within applications. The fundamental challenge that OPA addresses is the pervasive need for consistent and auditable policy enforcement across a heterogeneous landscape of services and platforms.

Traditionally, policy logic is often hardcoded directly into applications or scattered across various configuration files. This approach leads to several significant drawbacks:

1. Inconsistency: Different services or teams might implement similar policies in slightly different ways, leading to security gaps or unpredictable behavior.
2. Tight Coupling: Policy changes necessitate modifications and redeployments of application code, slowing down development cycles and increasing the risk of errors.
3. Lack of Centralization: Auditing and managing policies become difficult when they are embedded and fragmented, making it challenging to get a holistic view of an organization's security posture.
4. Limited Reusability: Policies developed for one service are typically not easily transferable to another, leading to redundant effort.

Open Policy Agent (OPA) was created to solve these problems by introducing the concept of policy as code. OPA acts as a lightweight, general-purpose policy engine that decouples policy decision-making from application logic. The core model is straightforward: a service queries OPA with a JSON input representing an event or request (e.g., a user trying to access a resource, a Kubernetes pod deployment). OPA evaluates this input against pre-loaded policies, written in its declarative policy language called Rego, and returns a JSON output—typically an allow/deny decision, but it can be any structured data.

Architecturally, OPA operates as a sidecar, host-level daemon, or library. It continuously loads policy bundles and relevant data, ensuring it has the most up-to-date information for decision-making. Crucially, OPA can also be configured to send decision logs to a central store, providing invaluable data for auditing, debugging, and monitoring policy enforcement. This standardization and decoupling allow developers to focus on core business value, while policy experts can manage and evolve policies independently, using familiar software development practices like version control, testing, and static analysis.

Key Findings

▶ Watch: How Open Policy Agent (OPA) works architecturally (2:20)

The talk highlighted several key findings and advancements, showcasing OPA's maturity, widespread adoption, and ongoing development:

1. Community Growth and Adoption: OPA has garnered significant traction, surpassing 10,000 GitHub stars since the last KubeCon, indicating a robust and engaged community. Public endorsements from major organizations like Bloomberg underscore OPA's effectiveness in decoupling access control logic and providing a generic, consistent policy enforcement mechanism across diverse services and systems.

1. Enhanced Developer Experience for Rego:

• Rego Debug Adapter Protocol (DAP) Support: A significant usability improvement, the Rego Linter Language Server now supports the Debug Adapter Protocol. This allows developers to use standard programming tools for working with Rego, enabling more effective debugging and iteration, similar to how they would debug other code languages.
• Upcoming Language Features (Roadmap): The OPA team is actively exploring new features to make Rego more accessible and powerful:
• Streaming Tests: To provide immediate feedback as test suites run, akin to go test, improving integration with language tooling and live feedback in editors.
• Logical OR Operator: Addressing a common user expectation and simplifying policy expression, which currently requires more verbose constructs.
• Alternative Operator: A ternary-like operator to handle undefined references gracefully, allowing policies to attempt an evaluation and fall back to a default value if the reference is missing, preventing entire rule failures.
• String Interpolation: A highly anticipated feature to simplify the creation of dynamic strings (e.g., deny messages) and mitigate issues where sprintf fails if a referenced variable is undefined, improving policy robustness.

1. OPA Gatekeeper Advancements: The sibling project, OPA Gatekeeper, which provides Kubernetes admission control using custom resources, has seen notable updates:

• OPA 1.0 Support: Gatekeeper now fully supports OPA 1.0, allowing users to leverage the latest Rego features by setting the source_version field.
• Pub/Sub Interface Improvements: Enhancements to the publishing and subscription mechanisms.
• Gator CLI Enhancements: The command-line tool for Gatekeeper users has received updates, improving its utility for pre-deployment policy validation.
• Violation Export to Disk: New capabilities for exporting policy violations, aiding in auditing and reporting.
• Validating Admission Policy Integration to Beta: Moving towards a more mature integration with Kubernetes' native validating admission policies.

1. OPA 1.0 Release: A major milestone released at the end of last year, OPA 1.0 consolidates and standardizes a new version of Rego. This release is accompanied by comprehensive upgrade documentation and a release blog post to guide users through the transition and highlight its significance.

1. Groundbreaking Performance Improvements: Anders Eknert detailed a focused effort on OPA performance, primarily driven by the needs of Regal, the Rego linter and language server, which itself is written in Rego.

• Motivation: Regal's extensive Rego codebase (around 100 lint rules, totaling 15,000 lines of Rego) and its requirement to lint files on every key press in an editor exposed performance bottlenecks. A typical scenario of linting 200 files means evaluating approximately 3 million lines of Rego.
• Results: Through optimizations like reducing memory allocations in Go and improving built-in function runtimes, OPA 1.x achieved a staggering 723% faster evaluation compared to OPA 0.70 for these heavy workloads. This dramatically improved the developer experience for Regal and benefits all OPA users by making policy evaluation significantly faster and more efficient.

These findings collectively paint a picture of OPA as a rapidly evolving, battle-tested project that is continuously improving its core engine, language features, and ecosystem tools to meet the demands of modern cloud-native environments.

Technical Deep Dive

▶ Watch: Key benefits of policy as code (4:10)

OPA's power lies in its elegant architecture and its declarative policy language, Rego. At its core, OPA functions as a policy decision point (PDP), externalizing authorization and policy logic from applications and services. When a service needs a policy decision, it sends a query to OPA. This query is a JSON document representing the context of the decision, such as a user's identity, the resource being accessed, or the properties of a new resource being deployed. OPA, acting as the policy enforcement point (PEP), evaluates this JSON input against a set of policies written in Rego and any relevant data it has loaded, returning a JSON output. This JSON-in, Rego-evaluation, JSON-out model provides a highly flexible and standardized interface.

Rego, OPA's domain-specific language, is a declarative query language based on Datalog. It allows users to define rules that operate on structured data. Policies are expressed as collections of rules that define what is allowed or denied. For example, a Rego policy might state that "a user can access a resource if they are in the 'admin' group and the resource is tagged as 'public'." The declarative nature means you describe what the policy should achieve, rather than how to achieve it, leaving the execution details to the OPA engine. This contrasts sharply with imperative languages, making policies more concise, auditable, and easier to reason about.

Beyond core policy evaluation, OPA's architectural design includes mechanisms for policy and data loading. OPA is configured to fetch policy bundles (collections of Rego policies and associated data) from a configured source, ensuring it always operates with the latest rules. Similarly, it can load external data (e.g., user roles, resource attributes) into its memory, making this context available for policy evaluation. Another critical component is the decision log store. If configured, OPA sends detailed records of every policy decision it makes to an external log, providing an invaluable audit trail for compliance, security analysis, and debugging.

One of OPA's most prominent use cases is Kubernetes admission control, facilitated by the OPA Gatekeeper project. Gatekeeper integrates OPA as a validating and mutating admission webhook in Kubernetes. This allows cluster administrators to define and enforce custom policies on resources entering the cluster. Policies are expressed as Kubernetes Custom Resources (CRs), making them manageable using standard Kubernetes tooling. For instance, Gatekeeper can enforce policies like "all container images must come from an approved registry," "no pods should run as root," or "all deployments must have specific labels." The gator CLI tool further enhances this by allowing Gatekeeper users to perform pre-deployment checks and validations of their policies, catching issues before they impact the cluster. The recent updates to Gatekeeper, including support for OPA 1.0 and improved pub/sub interfaces, underline its continuous evolution in the Kubernetes ecosystem.

The release of OPA 1.0 marked a significant milestone, standardizing the Rego language and the OPA runtime. This version ensures greater stability and predictability for users and developers. It also introduced the source_version field, allowing users to explicitly specify the Rego version their policies are written for, facilitating smoother upgrades and future compatibility.

A major focus of the talk was the performance optimization work driven by Anders Eknert, primarily motivated by the needs of Regal, the Rego linter and language server. Regal itself is written in Rego and comprises approximately 100 lint rules, totaling 15,000 lines of Rego. When Regal is used as a language server in an editor like VS Code, it needs to lint files frequently, potentially on every key press. Linting a moderately sized project of 200 Rego files means evaluating about 3 million lines of Rego for each linting pass. Initially, this could take 2-3 seconds on a modern MacBook Pro, which is too slow for real-time editor feedback and even slower in resource-constrained CI/CD environments where parallelization is limited.

The optimization strategy focused on typical Go application performance improvements:

1. Identifying Hot Paths: Pinpointing the most frequently executed code sections.
2. Reducing Memory Allocations: Minimizing the number of times the Go runtime needs to allocate memory, which is a common bottleneck.
3. Improving Built-in Function Runtimes: Optimizing the performance of OPA's approximately 180 built-in functions.
4. Data Structure Optimizations: Refining the underlying data structures used by OPA.

A concrete example of these optimizations is number interning within the count() function. Previously, every call to count() would allocate new memory for the resulting number. If count() was called millions of times, even for the same number, it would trigger millions of memory allocations. With interning, if the same number is requested multiple times, OPA now returns a reference to the same pre-allocated number, drastically reducing memory overhead and allocation time. These optimizations, implemented from OPA version 0.70 onwards and fully integrated into OPA 1.x, resulted in a remarkable 723% faster evaluation for Rego workloads, significantly benefiting Regal and all OPA users.

The future roadmap for OPA continues to enhance the developer experience. Streaming tests will provide immediate feedback during policy testing, similar to go test, by printing results as they become available rather than waiting for the entire suite to complete. The introduction of a logical OR operator aims to simplify common policy patterns that currently require more verbose workarounds, making Rego more intuitive for new users. Similarly, an alternative operator (like a ternary operator) would allow policies to gracefully handle cases where a referenced variable might be undefined, providing a default value instead of causing the entire rule to fail. Finally, string interpolation is poised to improve policy readability and robustness by allowing variables to be directly embedded in strings, solving the sprintf problem where undefined references can cause policy evaluation failures. These ongoing developments demonstrate OPA's commitment to being a powerful, performant, and user-friendly policy engine.

Demo / Proof of Concept

▶ Watch: Real-world OPA use cases and testimonials (6:00)

While the talk did not feature a live, interactive demonstration of OPA in action, it provided several conceptual proofs of concept and code examples to illustrate the impact of new features and performance improvements.

For instance, the concept of streaming tests was visually demonstrated by comparing the output of a standard go test run with the current opa test output. The go test example showed immediate, line-by-line feedback as tests passed or failed, whereas the opa test output only appeared after the entire test suite had completed. This visual comparison effectively conveyed the desired improvement for developers seeking quicker feedback loops when writing and testing Rego policies.

The discussion around string interpolation included a clear example of the problem it aims to solve. The speakers illustrated how using the sprintf built-in function to construct a deny message could fail if a referenced variable (e.g., user.age or groups.tester_group) was undefined. This would cause the entire rule to fail, even if the core policy logic was otherwise correct. They then conceptually showed how string interpolation would provide a more robust and readable way to construct such messages, potentially allowing for fallback values or simply omitting the undefined part, preventing unexpected policy failures.

Perhaps the most compelling "proof of concept" presented was the performance benchmark graph. This graph visually demonstrated the dramatic improvement in OPA's evaluation speed. It showed that for a workload of 10,000 lines of Rego, OPA version 0.70 took almost 2 seconds to evaluate. However, with the optimizations introduced in OPA 1.x, the evaluation time dropped to approximately 0.5 seconds, representing a 723% faster evaluation. This tangible data, presented as a clear bar chart comparing different OPA versions, served as a strong empirical proof of the success of the performance optimization efforts. The red bars on the graph also indicated a reduction in memory usage, further solidifying the impact of avoiding memory allocations on hot paths.

These examples, while not interactive, effectively communicated the practical benefits and technical advancements discussed throughout the presentation.

Defensive Implications

▶ Watch: OPA community updates and Rego debugger improvements (7:45)

The Open Policy Agent provides a foundational shift in how organizations can approach security and compliance, offering significant defensive implications for modern cloud-native environments:

1. Standardized and Consistent Policy Enforcement: OPA enables organizations to define security policies in a single, standardized language (Rego) and enforce them consistently across diverse systems – from Kubernetes clusters and API gateways to microservices and CI/CD pipelines. This eliminates policy drift and reduces the attack surface by ensuring that security rules are applied uniformly, preventing ad-hoc or inconsistent implementations that can lead to vulnerabilities.

1. Decoupled Security Logic: By externalizing policy decisions from application code, OPA allows security teams to manage and update policies independently of development cycles. This means security controls can be evolved and hardened rapidly without requiring application redeployments, significantly improving an organization's agility in responding to new threats or compliance requirements.

1. Policy as Code and GitOps for Security: Treating policies as code allows organizations to leverage standard software development practices for security. Policies can be version-controlled in Git, enabling pull request workflows for peer review, automated testing, and audit trails of all policy changes. This fosters collaboration between security and development teams and integrates security directly into the development pipeline, promoting a "shift-left" security approach.

1. Robust Kubernetes Admission Control (OPA Gatekeeper): OPA Gatekeeper offers powerful defensive capabilities for Kubernetes environments. It acts as an admission controller, intercepting requests to the Kubernetes API server and enforcing policies before resources are created or modified. This prevents the deployment of non-compliant or insecure configurations, such as:

• Disallowing containers running as root.
• Enforcing image provenance (e.g., only from approved registries).
• Requiring specific labels or annotations for auditing and management.
• Limiting resource requests and limits to prevent resource exhaustion attacks.
• Ensuring network policies are correctly applied.

This proactive enforcement significantly hardens the Kubernetes control plane and workloads.

1. Enhanced Auditability and Compliance: OPA's decision logging feature provides a comprehensive audit trail of every policy decision made. This log contains detailed context about the request, the policy evaluated, and the final decision. This data is invaluable for compliance reporting, forensic analysis during security incidents, and continuous monitoring of policy effectiveness.

1. Proactive Policy Validation with Regal: The Regal linter and language server, built in Rego, is a critical defensive tool. It helps policy authors identify errors, potential misconfigurations, and stylistic issues in their Rego policies before they are deployed. This proactive validation reduces the likelihood of deploying flawed policies that could inadvertently create security gaps or cause operational disruptions, thereby improving the overall quality and reliability of security controls.

1. Improved Performance for Critical Decisions: The significant performance improvements (723% faster evaluation) mean that OPA can make policy decisions much quicker. This is crucial for high-throughput systems where latency is a concern. Faster evaluation allows for more comprehensive and granular policy checks without introducing performance bottlenecks, ensuring that security decisions don't impede application responsiveness or user experience.

1. Future-Proofing Policies with Language Enhancements: Upcoming features like the logical OR operator, alternative operator, and string interpolation will make Rego policies easier to write, more robust, and less prone to errors related to undefined data. This improved developer experience contributes to stronger defenses by enabling security teams to write more sophisticated and reliable policies with fewer accidental loopholes.

In essence, OPA transforms policy enforcement from a fragmented, reactive process into a standardized, proactive, and auditable component of an organization's security architecture, making systems inherently more secure and compliant.

Key Takeaways

• Standardized Policy Enforcement: OPA enables organizations to define and enforce policies as code, providing a consistent and decoupled approach to authorization and governance across diverse services and platforms.
• OPA Gatekeeper for Kubernetes Security: Gatekeeper empowers Kubernetes users to enforce admission policies effectively, preventing insecure configurations and ensuring compliance within their clusters.
• OPA 1.0 Marks Project Maturity: The release of OPA 1.0 signifies a stable and standardized version of the Rego language and runtime, making it a robust choice for production environments.
• Dramatic Performance Gains: Recent optimizations have made OPA significantly faster (723% for certain workloads) and more memory-efficient, improving developer experience and enabling more complex policy evaluations without performance overhead.
• Enhanced Developer Experience for Rego: Tools like the Rego Debug Adapter Protocol (DAP) support and upcoming language features (logical OR, string interpolation) are continuously improving the usability and robustness of Rego policy development.
• Policy as Code Drives Security: By treating policy as code, organizations can leverage version control, static analysis (with Regal), and automated testing to build more secure, auditable, and resilient systems.

About the Speaker(s)

Charlie Egan is a maintainer of the Open Policy Agent (OPA) project and works on the developer relations team at Styra. His role involves engaging with the OPA community, educating users, and contributing to the project's development and outreach.

Anders Eknert is also a maintainer of the Open Policy Agent (OPA) project and a member of the developer relations team at Styra. He has a particular focus on OPA performance, leading significant efforts to optimize the engine's speed and efficiency, especially for demanding workloads like the Regal linter. His work is crucial in ensuring OPA remains performant for its growing user base.