Compliance at the Speed of Innovation: Leveraging AI-Driven Automation for Real-Time Regulatory Read

Compliance at the Speed of Innovation: Leveraging AI-Driven Automation for Real-Time Regulatory Read

Speakers: Larry Kurvalo (Moderator), Robert Fkala (Sunstone Secure), Simon Metson (EDB), Anka Syler (Red Hat, IBM), Eugi Watan (IBM Research Tokyo)

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=DmfZq70WOxI

Overview

This KubeCon EU session, "Compliance at the Speed of Innovation: Leveraging AI-Driven Automation for Real-Time Regulatory Read," explored the critical shift from traditional, manual compliance processes to automated, AI-driven approaches in the context of rapidly evolving cloud-native environments. Moderated by independent analyst Larry Kurvalo, the panel featured industry experts Robert Fkala, Simon Metson, Anka Syler, and Eugi Watan, who collectively illuminated the challenges and opportunities in achieving continuous compliance through standardization and automation. The discussion highlighted the necessity of adapting compliance strategies to the dynamic nature of modern infrastructure, emphasizing the role of open-source initiatives like Compliance as Code and AI technologies in this transformation.

The talk underscored a fundamental tension: the increasing volume and complexity of regulations (such as DORA and the EUI for AI) and the industry's move towards continuous compliance, juxtaposed with the inherent dynamism and ephemeral nature of cloud-native systems. Traditional compliance methods, often relying on static documents and manual audits, are proving insufficient to meet these demands. The panelists converged on the idea that artificial intelligence, particularly Generative AI (GenAI) and intelligent agents, offers a powerful pathway to bridge this gap, enabling organizations to achieve real-time regulatory readiness and manage compliance as a scalable, engineering problem rather than a static, administrative burden.

Ultimately, the session aimed to provide a roadmap for organizations to navigate this complex landscape, advocating for a cultural shift within compliance teams and the adoption of technical frameworks that allow compliance to keep pace with the speed of innovation. It delved into specific open-source projects, AI methodologies, and strategic considerations for integrating these advanced capabilities into enterprise security and governance practices, setting the stage for a future where compliance is not an impediment but an intrinsic, automated component of the development lifecycle.

Background

▶ Watch: Introduction to compliance as code and continuous compliance. (0:00)

The landscape of regulatory compliance has historically been characterized by arduous, manual processes. Organizations typically faced annual or quarterly audits, relying on static PDF documents, spreadsheets, and manual evidence collection—a process often described as "paper-driven." This approach, while perhaps manageable in more monolithic, stable IT environments, has become untenable with the advent of cloud-native architectures and the rapid pace of digital transformation. Simon Metson vividly illustrated this challenge, recalling an engineer spending an entire day manually pulling lists of servers and ports for an audit, highlighting the inefficiency and human toll of traditional methods.

The core problem stems from several factors. Firstly, the sheer volume and complexity of new regulations are escalating globally, exemplified by directives like DORA (Digital Operational Resilience Act) and the EU AI Act. These regulations demand a more agile and continuous approach to compliance, moving away from infrequent snapshots to real-time monitoring and enforcement. Secondly, the dynamism of cloud-native environments, characterized by ephemeral workloads, auto-scaling, and developer-driven provisioning, fundamentally clashes with the static inventory and change control assumptions of many traditional compliance frameworks (e.g., FedRAMP or NIST 800-53). A system where thousands of instances can be spun up or down via an API within minutes challenges the notion of a general signing off on every change.

To address these issues, the concept of Compliance as Code has emerged as a critical paradigm shift. This involves expressing compliance requirements, controls, and policies in machine-readable formats, allowing for automation, versioning, and integration into the software development lifecycle. A cornerstone of this movement is Oscal (Open Security Control Assessment Language), a NIST standard. Anka Syler explained that Oscal provides schemas for specifying control catalogs and mapping these controls to technology-specific rules, check IDs, and associated evidence. This programmatic approach aims to transform abstract regulatory text into actionable, verifiable code.

Prior work in this space has also seen the rise of Policy as Code, utilizing tools like Ansible, Open Policy Agent (OPA), and Kyverno to define and enforce security and operational policies programmatically. However, even with these advancements, the manual effort required to translate complex natural language regulations into code, identify gaps, and maintain continuous compliance across diverse, rapidly changing environments remained significant. This persistent challenge laid the groundwork for the panel's discussion on how AI, particularly GenAI and intelligent agents, could further accelerate and automate these processes, pushing the boundaries of what "real-time regulatory readiness" truly means.

Key Findings

▶ Watch: Addressing manual audit pain with open-source automation. (2:10)

The talk presented several key findings and insights into the evolving landscape of compliance, particularly concerning the integration of AI and the challenges of cloud-native environments:

1. AI as a Compliance Accelerator: The panelists unequivocally demonstrated that AI, especially Generative AI (GenAI) and intelligent agents, can significantly accelerate various stages of the compliance lifecycle. Eugi Watan highlighted AI's ability to analyze natural language policy documents, extract information, map it to existing compliance frameworks, identify gaps, and even generate new controls or policy changes. This drastically reduces the manual effort and time involved in understanding and adapting to new regulations.

1. The Power of Oscal and Oscal Compass: Oscal (Open Security Control Assessment Language) was presented as a foundational NIST standard for Compliance as Code. Anka Syler detailed how Oscal provides standardized schemas for control catalogs, rule expression for technology, and mapping to check IDs and evidence. The CNCF Oscal Compass project, which includes an SDK (Trestle) and a platform, was identified as a critical open-source initiative for managing these compliance layers programmatically, fostering collaboration and standardization.

1. Cloud-Native Dynamism vs. Static Compliance: Simon Metson articulated the core conflict between the inherent dynamism of cloud-native environments (e.g., ephemeral pods, auto-scaling) and the static, inventory-based assumptions of traditional compliance frameworks. While this dynamism poses significant challenges for evidence collection and change management, the declarative nature of Kubernetes offers a key opportunity. By provisioning resources through operators, consistency can be guaranteed, making it easier to "stamp out" approved configurations.

1. The Emergence of AI-Native Compliance Challenges: Robert Fkala introduced the concept of "AI-native compliance," where AI agents not only analyze but also generate code, deployments, and even container images. This scenario creates an "orders of magnitude more dynamic" environment, shifting the human task from crafting small code blocks to reviewing vast amounts of AI-generated code. This raises critical questions about human trust loops, skill sets for evaluation, and the scalability of human oversight. The example of Google using LLMs to generate a thousand controllers underscores this shift.

1. Benchmarking and Auditability of AI Agents: A crucial finding was the necessity of ensuring the trustworthiness and correct operation of AI used for compliance. Anka Syler mentioned the IBM IT bench as an open-source solution for benchmarking AI agents against 50 compliance scenarios (e.g., CIS benchmarks) deployed in real environments using tools like Ansible, OPA, and Kyverno. Eugi Watan emphasized agent explainability—recording the agent's trajectory and providing transparency—as a key approach to address the auditability challenge in regulated scenarios.

1. Cultural Change as the Longest Pole: Anka Syler stressed that while technological solutions are advancing rapidly, the biggest hurdle to adopting Compliance as Code and AI-driven compliance is cultural change within compliance teams. These teams are often non-technical, accustomed to manual processes, and can be overwhelmed by new technologies, agents, and AI. The analogy to infrastructure as code's 10-15 year maturation period suggests a similarly long journey for compliance as code, with cultural adaptation being the most time-consuming aspect.

1. Rapid Time to Value for AI in Discovery: Robert Fkala noted that even if "perfection" in AI-native compliance is years away, organizations can achieve rapid "time to value" by using AI for initial assessment and discovery. LLMs can ingest and vectorize existing bespoke documentation, tribal knowledge, and mixed infrastructure details, often surfacing "unknown unknowns" and components that even security teams were unaware of.

Technical Deep Dive

▶ Watch: Oscal: NIST standard for compliance as code framework. (3:40)

The session provided a deep dive into the technical underpinnings of Compliance as Code and the application of AI, highlighting specific tools, standards, and methodologies.

At the heart of the Compliance as Code movement is Oscal (Open Security Control Assessment Language), a NIST standard. Anka Syler explained Oscal's layered approach:

• Control Catalog Schemas: Oscal provides standardized schemas for defining compliance controls. This moves away from ambiguous natural language in PDFs to a structured, machine-readable format.
• Technology-Specific Rules: It allows for expressing how these controls manifest as rules within specific technologies. For instance, a control requiring encryption at rest could be translated into a rule for a specific database or storage system.
• Mapping to Check IDs and Evidence: Oscal enables mapping these rules to concrete check IDs that can be programmatically verified, and to the evidence associated with their implementation. This creates a clear, auditable trail.

To operationalize Oscal, the CNCF Oscal Compass project plays a crucial role. This project provides an SDK called Trestle, which facilitates the management and manipulation of Oscal artifacts. Additionally, it offers a platform built on top of the SDK, designed to help organizations manage all the layers specified by Oscal, from high-level regulations down to specific technical checks. This open-source initiative aims to standardize and accelerate the adoption of Compliance as Code across the industry.

The integration of AI and Generative AI (GenAI) forms the cutting edge of this technical evolution, particularly in addressing the challenges of natural language and dynamic environments. Eugi Watan detailed a layered approach for AI application:

1. Natural Language Processing (NLP): Initial input documents, often abstract and non-operational (like regulatory text), are first processed by AI to extract key information.
2. Retrieval Augmented Generation (RAG): To provide context and accuracy, AI systems utilize RAG mechanisms. This involves retrieving relevant, curated bodies of documents—such as existing compliance frameworks, controls, and requirements—and feeding them to the AI. This helps the AI perform crosswalking, identifying commonalities between multiple regulations and mapping new requirements to existing controls.
3. Gap Analysis and Policy Generation: If a new regulation introduces requirements not covered by existing controls, GenAI is employed to identify these gaps and generate the necessary new policies or controls. Similarly, if existing policies require changes, AI agents can identify the "delta requirements" and suggest modifications to bring them into alignment.
4. Agent-Driven Automation: Intelligent agents are designed to navigate the entire compliance process. They can detect new requirement changes or control updates, then navigate relevant stakeholders (human or automated) to take action. This can be integrated into CI/CD pipelines to shift compliance activities "left," making them a continuous part of the development and deployment process.

A significant challenge in using AI for compliance is ensuring its trustworthiness and auditability. Anka Syler introduced the concept of benchmarking AI agents, citing the IBM IT bench as an example. This open-source platform provides approximately 50 compliance scenarios, often based on CIS benchmarks, which are deployed in real environments. These scenarios leverage tools like Ansible, Open Policy Agent (OPA), and Kyverno to test agent behavior. The IT bench includes "ground truth" for each scenario, allowing for objective comparison and evaluation of how different agents perform against a known correct answer.

For auditability, Eugi Watan emphasized agent explainability. This involves meticulously recording the entire trajectory of an AI agent's operations—every decision, every piece of data processed, and every output generated. This record provides transparency for human verifiers and serves as crucial evidence for later compliance audits, addressing concerns about the "black box" nature of some AI systems in regulated environments.

Finally, the discussion touched upon the dynamic nature of AI-native compliance, where Large Language Models (LLMs) are not just analyzing but actively generating code. Robert Fkala cited an example from Google where LLMs generated a thousand controller codes, highlighting a shift in complexity from manual coding to reviewing and validating AI-generated artifacts. This dynamic environment, where images and deployments might be generated on the fly, further complicates traditional static inventory models and necessitates continuous, automated validation. The challenge of patching CVEs in immutable images within strict timelines (e.g., 24 hours) exemplifies the need for an extremely agile, AI-assisted approach to maintain security and compliance at scale.

Demo / Proof of Concept

▶ Watch: Applying AI, GenAI, and agents for regulatory compliance. (5:40)

The talk did not feature a live, interactive demonstration of a specific new tool or a direct proof-of-concept for a novel AI-driven compliance solution. Instead, the panelists focused on elucidating conceptual frameworks, discussing existing open-source projects, and outlining the theoretical and practical application of AI to compliance challenges based on their research and development efforts.

However, the session did refer to key components that serve as foundational elements for such proofs-of-concept. Anka Syler mentioned the IBM IT bench, an open-source platform designed for benchmarking AI agents. This platform includes about 50 compliance scenarios, often based on CIS benchmarks, which are deployed in real environments using tools like Ansible, Open Policy Agent (OPA), and Kyverno. While not a live demo during the presentation, the IT bench effectively functions as a proof-of-concept environment where organizations can "subscribe their own agents" or utilize existing ones to test their performance against established compliance requirements and known ground truths. This provides a structured way to evaluate the trustworthiness and effectiveness of AI in a controlled, realistic setting, which is a critical step before broader deployment.

Similarly, the CNCF Oscal Compass project was presented as a practical implementation of Compliance as Code. With its Trestle SDK and platform, Oscal Compass provides the tools and framework necessary to manage compliance artifacts programmatically. While no direct demonstration of Oscal Compass was given, its existence and ongoing development underscore the availability of open-source components that form the backbone of the AI-driven compliance vision articulated by the panelists.

Defensive Implications

▶ Watch: Ensuring trustworthy AI for compliance using benchmarking. (7:50)

The insights shared by the panelists carry significant defensive implications for organizations aiming to strengthen their security posture and achieve continuous compliance in cloud-native and AI-driven environments.

1. Embrace Compliance as Code (CaC) and Policy as Code: Defenders must move away from manual, document-driven compliance to programmatic approaches. Adopting frameworks like NIST Oscal and leveraging open-source projects such as CNCF Oscal Compass (with its Trestle SDK) is crucial. This enables machine-readable definition of controls, automated checks, and integration into CI/CD pipelines, shifting compliance left in the development lifecycle. Tools like OPA and Kyverno are essential for enforcing Policy as Code in Kubernetes environments.

1. Strategically Integrate AI/GenAI: Defenders should explore using AI for initial policy analysis, crosswalking regulatory frameworks, and identifying gaps. AI agents can significantly reduce the manual burden of ingesting new regulations and mapping them to existing controls. This allows security teams to focus on higher-value tasks, like threat modeling and architectural security, rather than tedious document review. Robert Fkala's point about AI surfacing "unknown unknowns" highlights its value in discovering latent risks and undocumented assets.

1. Prioritize Declarative Infrastructure: For cloud-native environments, defenders should advocate for and implement declarative infrastructure through tools like Kubernetes operators. As Simon Metson explained, this ensures consistency in provisioning and configuration, making it easier to verify compliance. Once an operator-managed service is approved, subsequent deployments inherit that approved state, simplifying audits and reducing configuration drift.

1. Invest in Cultural Change and Upskilling: The biggest defensive challenge is not technical but cultural. Security leaders must actively work to bridge the gap between non-technical compliance teams and technology. This involves providing training, support, and potentially integrating compliance functions more closely with technical teams (e.g., DevOps, SREs). The goal is to turn compliance problems into "software development engineering problems" that can be solved with code and automation, rather than remaining administrative burdens.

1. Demand AI Agent Trustworthiness and Explainability: As AI agents become integral to compliance, defenders must ensure these agents are trustworthy and auditable. Implement rigorous benchmarking processes, similar to the IBM IT bench, to validate agent performance against known compliance scenarios and ground truths. Furthermore, require agent explainability—the ability to record and transparently present an agent's decision-making process—to provide crucial evidence for internal and external audits and to build human confidence in automated systems.

1. Prepare for Hyper-Dynamic Environments: The shift to AI-generated code, deployments, and even container images means environments will become "orders of magnitude more dynamic" (Robert Fkala). Defenders must adapt their monitoring, incident response, and vulnerability management strategies to cope with this speed. This includes automating CVE patching processes for ephemeral immutable images and developing rapid, AI-assisted threat detection capabilities.

1. Maintain Human Judgment in the Loop: Despite the power of AI, human judgment remains indispensable, especially for high-stakes decisions like risk acceptance and incident response. CISOs and security leaders will continue to provide the "gravitas and judgment" (Robert Fkala) in moments of crisis, interpreting AI outputs, and making business-critical decisions that LLMs cannot fully replicate. Defensive strategies must therefore focus on augmenting human capabilities with AI, not replacing them entirely.

Key Takeaways

• Compliance Must Automate to Keep Pace: The traditional, manual compliance model is unsustainable given the increasing volume of new regulations (e.g., DORA, EU AI Act) and the industry's shift towards continuous compliance. AI-driven automation is essential to achieve real-time regulatory readiness and keep pace with the speed of innovation in cloud-native environments.
• Oscal and Oscal Compass are Foundational: NIST Oscal (Open Security Control Assessment Language) provides a critical open standard for defining compliance controls programmatically. The CNCF Oscal Compass project, with its Trestle SDK, offers the open-source tools and platform necessary to manage these Compliance as Code layers effectively.
• AI/GenAI Agents Drive Efficiency, But Need Scrutiny: AI, especially Generative AI and intelligent agents, can significantly accelerate compliance by automating natural language analysis, cross-framework mapping, gap identification, and even policy generation. However, their trustworthiness and correct operation must be ensured through rigorous benchmarking (e.g., IBM IT bench) and agent explainability for auditability.
• Cloud-Native Dynamism is a Double-Edged Sword: While the ephemeral and auto-scaling nature of cloud-native environments poses significant challenges for static compliance models, the declarative nature of Kubernetes offers a powerful opportunity for consistent, pre-approved configurations, simplifying evidence collection and continuous validation.
• Cultural Change is the Hardest and Longest Challenge: The most substantial barrier to widespread adoption of Compliance as Code and AI-driven compliance is the necessary cultural transformation within often non-technical compliance teams. This shift will take years and requires significant investment in upskilling and integrating compliance into engineering workflows.
• Human Judgment Remains Indispensable: Despite the advancements in AI, human judgment, expertise, and the ability to make nuanced business risk decisions will remain critical for CISOs and security leadership, particularly in crisis situations and for establishing trust in AI-driven systems.

About the Speaker(s)

• Larry Kurvalo (Moderator): An independent analyst who moderated the panel discussion, guiding the conversation around compliance as code and the role of AI.
• Robert Fkala: With Sunstone Secure, a company he started in 2019, Robert focuses on public sector, healthcare, and heavily regulated industries like pharma. His work centers on managing compliance in large enterprises with minimal staff and budget, applying knowledge graphs and machine learning (specifically graph RAG) over the past five years. He is involved in the Kubernetes policy workgroup, formal verification with NIST, and contributes AI-related security and compliance insights to nativesecurity.org.
• Simon Metson: As SVP of Engineering at EDB, Simon's work primarily involves PostgreSQL, data storage, processing, AI, and analytics. With over 13 years of experience in cloud products, he has led teams through SOC 2 and PCI audits, driving the automation of compliance problems. He open-sourced the project Auditry five to six years ago and is actively working with the Oscal and Compass projects, exploring their integration into the CNCF.
• Anka Syler: A Distinguished Engineer at Red Hat and IBM, Anka is a key figure in the Compliance as Code movement. She is deeply involved with Oscal (Open Security Control Assessment Language), a NIST standard for specifying control catalogs, rules, and evidence. She also plays a significant role in the CNCF Oscal Compass project, which provides an SDK (Trestle) and a platform for managing these compliance layers.
• Eugi Watan: From IBM Research Tokyo, Eugi's expertise lies in applying AI, Generative AI (GenAI), and agent technology to the policy as code field. His research focuses on using AI to analyze existing policy documents, map requirements, identify gaps, and generate controls, aiming to improve the overall speed and efficiency of the compliance process.