RAGe Against the Machine: AI-Powered Compliance and Cybersecurity

RAGe Against the Machine: AI-Powered Compliance and Cybersecurity

Speakers: Brennan Lodge, Security Expert, Fractional CISO

Conference: DefCamp

YouTube: https://www.youtube.com/watch?v=nEfzK8dZLog

Overview

In an era where cybersecurity threats are escalating and regulatory landscapes are becoming increasingly complex, Brennan Lodge’s DefCamp talk, "RAGe Against the Machine: AI-Powered Compliance and Cybersecurity," offers a compelling vision for leveraging artificial intelligence to empower defensive strategies, particularly within Governance, Risk, and Compliance (GRC). Lodge introduces Retrieval Augmented Generation (RAG) as a transformative, secure, and cost-effective AI technique designed to combat the overwhelming volume of alerts, vulnerabilities, and regulations that plague modern security teams.

The presentation critically examines the current state of AI adoption in cybersecurity, highlighting both its immense potential to democratize knowledge and streamline operations, as well as the pervasive challenges posed by vendor opacity, privacy concerns, and the notorious "hallucinations" of large language models (LLMs). Lodge champions a human-in-the-loop approach, positioning AI as a crucial "information ally" rather than an autonomous guard. Through practical examples and a demonstration of his open-source tool, Audit Caddy, he illustrates how organizations can "roll their own" AI solutions, gaining greater control over their data, intellectual property, and compliance posture.

This talk is particularly pertinent for security professionals, GRC practitioners, and decision-makers grappling with the dual pressures of maintaining robust security defenses and navigating an ever-expanding web of legal and industry mandates. By advocating for secure, transparent, and self-hosted AI implementations, Lodge provides a strategic roadmap for organizations to harness AI's power without compromising on security or cost-efficiency. His insights underscore the urgent need for a more analytical, engineering-driven approach to GRC, moving beyond reactive measures to proactive, AI-driven intelligence.

Background

▶ Watch: Why AI is desperately needed in cybersecurity (3:58)

The cybersecurity landscape is characterized by an unrelenting deluge of challenges. Security Operations Centers (SOCs) are inundated with alerts, new vulnerabilities emerge daily, and attackers are increasingly harnessing AI themselves, creating an asymmetric battleground. This constant pressure, coupled with a persistent shortage of skilled personnel, often leaves junior security analysts without the vital mentorship needed to navigate complex threat vectors and frameworks like MITRE ATT&CK. Simultaneously, the regulatory environment is experiencing an explosion of new laws and compliance mandates, particularly in Europe, but also steadily increasing in the US, covering everything from data privacy (e.g., GDPR) to age verification and specific AI governance. This complex web of legislation, often penned by politicians and lawyers, presents significant implementation and integration hurdles for enterprises.

Against this backdrop, the adoption of AI in cybersecurity is fraught with both promise and peril. Many vendors, in what Lodge describes as "sprinkling in AI everywhere," offer opaque solutions where the underlying mechanisms, cost models, and efficacy remain unclear. The shift to "tokens" as a pricing metric for AI services further complicates ROI justification, reminiscent of early cloud computing's transparency issues. Moreover, the inherent privacy risks of sending sensitive organizational data to public AI models for training—data that could inadvertently resurface in other users' responses—are a major deterrent for security-conscious organizations. CISOs, facing budget cuts and increased scrutiny, demand AI solutions that are demonstrably secure, timely in their deployment and response, and more cost-efficient than existing automation. Many current LLMs, while powerful, often "hallucinate" or provide responses that are not verifiable, a critical flaw in security and compliance contexts where accuracy and explainability are paramount. Lodge emphasizes that AI is not a new threat vector but a new technology; existing safeguards, much like those for SQL injection, can be adapted for prompt injection and other AI-specific vulnerabilities.

The traditional GRC function, as Lodge notes, is often perceived as an "ugly baby"—a necessary but frequently frustrating and reactive endeavor. Organizations typically face audits with policies hastily assembled, spread across disparate systems like Jira, SharePoint, and file repositories. This fragmented approach makes proactive gap analysis and continuous compliance nearly impossible, leading to a scramble for evidence every audit cycle. The industry relies heavily on expensive tools and auditors, with executives often "throwing money at it" without clear transparency or value. This creates a critical need for a paradigm shift, one where AI can provide the necessary leverage to transform GRC from a reactive, costly, and manual process into an agile, proactive, and intelligent function.

Key Findings

▶ Watch: GRC use case workflow with RAG and MCP (18:00)

Brennan Lodge's presentation unveils several key findings that underscore the transformative potential of AI in cybersecurity and GRC, particularly through the lens of Retrieval Augmented Generation (RAG):

• AI as a Co-Intelligence Ally, Not a Full Guard: Lodge strongly advocates for viewing AI as an "information ally" that guides human decision-making, rather than an autonomous guardian. He emphasizes the critical need for human-in-the-loop mechanisms, checks and balances, and a thorough understanding of threats. This co-intelligence approach, as described by Ethan Malik, democratizes access to information, consolidates tools, and provides actionable insights without independent action.
• RAG as a Secure and Cost-Effective AI Technique: The core finding is that Retrieval Augmented Generation (RAG) offers a robust and secure method for leveraging AI. Unlike generic LLMs that can hallucinate or leak sensitive data, RAG allows organizations to utilize their own, latest, and most relevant data for responses. This capability to keep data on-prem or within a Virtual Private Cloud (VPC) directly addresses critical privacy and intellectual property concerns, making it a "really good security AI technique."
• Self-Hosting RAG for Economic and Security Benefits: Lodge demonstrates that "rolling your own" RAG solution can be more cost-effective and secure than relying solely on public LLM APIs. He provides a benchmark of approximately $500 per month for a self-hosted RAG MVP using a dedicated GPU instance on AWS, processing 20 questions daily from five analysts (around 100,000 tokens). This compares favorably to Open AI's API, which was about $100 more expensive at the time of his benchmark, not accounting for the intangible but significant cost of intellectual property leakage.
• AI Drives Efficiency in GRC Operations: The talk highlights AI's ability to significantly accelerate and improve GRC processes. This includes performing rapid gap analysis between internal policies and new regulations, automating policy generation based on company specifics, and providing continuous compliance mapping against frameworks like ISO 27001, NIST CSF, and SOC 2. This shifts GRC from a reactive, audit-driven cycle to a proactive, real-time monitoring paradigm.
• Regulations are Grounded, and AI Can Map Them: Lodge's research reveals that the multitude of global privacy and cybersecurity laws, despite their varied wording, are often conceptually grounded in established frameworks like ISO 27001, NIST CSF, and SOC 2. AI, specifically fine-tuned BERT models (e.g., Roberta), can effectively classify and map these complex regulations to specific controls, providing a foundation for understanding and implementing compliance requirements more efficiently.
• Machine Comprehension Protocol (MCP) for Secure Interoperability: The presentation introduces the Machine Comprehension Protocol (MCP) as a vital intermediary for securely routing communications between hosts, clients, and AI models. MCP facilitates the interconnectivity of various internal and external data sources, ensuring that AI models can access relevant information while adhering to defined communication standards, despite its own evolving set of vulnerabilities.
• Open-Source Tools Empower GRC Automation: Lodge's Audit Caddy project, an open-source tool, serves as a tangible proof-of-concept for AI-powered GRC. It provides policy templates, "starter packs" (regulation-to-framework tables), and uses BERT models to classify policy documents against compliance criteria, generate new policies, and produce audit reports with confidence levels. This democratizes access to sophisticated GRC automation.

Technical Deep Dive

▶ Watch: Audit Caddy demo: Policy generation and reporting (30:00)

The technical core of Brennan Lodge's presentation revolves around Retrieval Augmented Generation (RAG), a sophisticated AI technique designed to address the limitations of traditional Large Language Models (LLMs) in cybersecurity and GRC.

At its heart, RAG operates on an analogy of a highly organized library. Instead of an LLM relying solely on its pre-trained knowledge base, RAG enables it to "look up" relevant information from a curated, external knowledge source. This process involves three primary components:

1. Sentence Embedding Model: This component acts like a highly intelligent indexer. It takes raw text data (documents, policies, regulations, internal reports) and converts them into numerical representations called vectors or embeddings. These vectors capture the semantic meaning and context of the text, allowing for more nuanced comparisons than simple keyword searches. The model then "upserts" these vectorized documents, often divided into smaller, semantically coherent chunks, into a vector database.
2. Vector Database: This is the "organized stack of books" in the analogy. Unlike traditional databases that store and retrieve data based on exact matches or simple indexing, a vector database excels at semantic search. When a user queries the system, the query is also converted into a vector. The database then rapidly finds and retrieves documents or chunks whose vectors are "closest" in meaning to the query's vector, even if they don't share exact keywords. This enables searching by context, themes, and understanding, which is crucial for complex GRC queries.
3. Large Language Model (LLM): This is the "librarian." Once the vector database retrieves the most relevant documents, these documents are provided as additional context to the LLM. The LLM then synthesizes this retrieved information with its own general knowledge to generate a tailored, accurate, and contextually relevant response. This process significantly reduces the risk of hallucinations (fabricating information) common in LLMs that rely solely on their internal, often outdated, training data.

A key advantage of RAG, particularly for security, is the ability to self-host the entire system. This means organizations can keep their proprietary data, models, and processing entirely within their own network, a Virtual Private Cloud (VPC), or on-premises infrastructure. This mitigates the significant privacy and intellectual property risks associated with sending sensitive data to third-party public LLM APIs, where it could potentially be used for training or even inadvertently exposed. Lodge benchmarked an MVP of a self-hosted RAG solution at approximately $500 per month using a relatively inexpensive Nvidia GPU instance on AWS, demonstrating its cost-effectiveness compared to the higher costs and IP risks of public APIs. This cost was based on 20 daily questions from five analysts, equating to around 100,000 tokens.

To orchestrate the communication between different components and potentially integrate with external data sources, Lodge discusses the Machine Comprehension Protocol (MCP). MCP acts as an intermediary "dongle," facilitating secure and standardized communication between a host (e.g., an application making a request), a client (orchestrating the request), and a server (housing the AI model and data). While acknowledging that MCP, like any protocol, has its vulnerabilities, Lodge emphasizes its utility in providing a structured framework for routing calls to various AI models (e.g., Claude, OpenAI, or internal models) and intertwining disparate internal and external data sources securely. This protocol handles the complexities of communication, allowing developers to focus on the application logic rather than building bespoke interoperability layers.

In the context of GRC, Lodge details how AI can perform sophisticated gap analysis and policy classification. He utilized BERT models, specifically a fine-tuned version called Roberta, known for its strength in sentence completion and text classification. His research involved training these BERT models against a vast corpus of data privacy laws from the UN, classifying them against foundational frameworks like ISO 27001, NIST CSF, and SOC 2. This allows the system to determine the percentage matches or confidence levels of an organization's internal policies (e.g., employee handbooks, information security policies) against specific control criteria within these frameworks.

The Audit Caddy tool, an open-source project developed by Lodge, exemplifies these technical principles. It includes:

• Policy Templates and Starter Packs: Over 50 policy templates and pre-built tables that map regulations (like the New York City Shield Act) to standard cybersecurity frameworks, providing a structured starting point.
• Two BERT Analyzers: Specialized BERT models capable of classifying uploaded documents against various compliance controls.
• Vector Database Integration: Policies and regulatory documents are upserted into a vector database, enabling semantic search and retrieval.
• Policy Generation Engine: A prompt engineering interface that, based on a survey of company-specific details (e.g., cloud instances, data residency, logo for watermarking), generates new policies (e.g., a privacy communications policy) adhering to typical policy frameworks. Lodge stresses that these are drafts requiring human review and customization.
• Compliance Reporting: The system can generate scorecards and detailed reports for frameworks like SOC 2, illustrating compliance progress, identifying risks, and prioritizing actions based on the confidence levels derived from policy classification. This moves GRC towards a data-driven, engineering-centric approach.

By combining RAG for secure data retrieval, MCP for robust communication, and fine-tuned BERT models for intelligent classification, Lodge demonstrates a powerful, transparent, and controllable AI architecture for managing the complexities of modern cybersecurity and compliance.

Demo / Proof of Concept

▶ Watch: Key recommendations for implementing AI in cybersecurity (33:20)

Brennan Lodge provided a practical demonstration of his open-source tool, Audit Caddy, which is available on GitHub and also features a free-for-individual-use UI at auditcaddy.com. The demo showcased how RAG and fine-tuned BERT models can dramatically streamline GRC tasks.

The demonstration began with Document Classification. Lodge uploaded a standard "Business Continuity and Disaster Recovery" policy document into Audit Caddy. On the backend, a specialized BERT model processed the document. The UI then displayed a quick and easy mapping of the uploaded policy to relevant SOC 2 control criteria, complete with confidence levels. This immediate classification allows organizations to quickly ascertain which controls are covered by their existing documentation, serving as direct evidence for auditors and significantly reducing the manual effort of compliance mapping across the 62 different controls typically managed in SOC 2.

Next, the demo illustrated Policy Search and Retrieval using the RAG capabilities. Lodge queried the system, asking, "Do we have a business or disaster recovery policy?" The backend, leveraging the vector database, quickly retrieved the relevant document. The system not only confirmed its existence but also provided a quick summary of the document's contents and the option to download it. This functionality empowers analysts to rapidly access specific policy information without sifting through countless files or relying on traditional keyword searches, which often miss contextual nuances.

A particularly innovative aspect of the demo was AI-Powered Policy Generation. Lodge demonstrated how Audit Caddy can create new policies tailored to a company's specific needs. By filling out a brief survey with company details—such as the type of business, cloud instances used, data residency concerns, and even a logo for watermarking—the system generated a new "Privacy Communications Policy." This generated policy included a proper framework and the specified watermark. Lodge emphasized that while AI can rapidly draft policies, it is not an "easy button"; human review, customization (e.g., adding specific email addresses), and justification are still crucial to ensure accuracy and alignment with organizational specifics. This feature drastically cuts down the initial drafting time for new policies, especially in response to emerging regulations.

Finally, Lodge showcased Compliance Report Generation. For a selected framework, such as SOC 2, Audit Caddy generated a "scorecard." This report provided a comprehensive overview of the company's compliance progress, highlighting areas of high confidence, medium confidence, and low confidence. The report could then be downloaded and shared with auditors or management, offering a transparent and data-backed justification of the organization's compliance posture, including identified risks and priorities. This transforms the often arduous task of preparing audit reports into an efficient, AI-assisted process.

Through these demonstrations, Lodge effectively proved the practical utility of RAG and specialized AI models in making GRC more efficient, transparent, and manageable, moving beyond theoretical concepts to tangible, actionable solutions.

Defensive Implications

▶ Watch: Q&A: Gaining executive buy-in for cybersecurity (43:00)

Brennan Lodge's talk provides critical insights and actionable strategies for defenders navigating the complex interplay of AI, cybersecurity, and compliance. The primary defensive implications revolve around strategic adoption of AI, demanding transparency, and shifting towards proactive, engineering-driven GRC.

1. Prioritize Secure AI Implementations with RAG: The most significant defensive implication is the strong recommendation to adopt Retrieval Augmented Generation (RAG). Defenders should prioritize RAG because it allows organizations to maintain complete control over their sensitive data and intellectual property. By storing data in vector databases within their own VPCs or on-premises environments, organizations mitigate the risks of data leakage and privacy breaches associated with sending proprietary information to public LLM APIs. This approach also significantly reduces the likelihood of AI hallucinations, as responses are grounded in verifiable, internal data. RAG is presented as a secure, cost-effective, and transparent starting point for AI adoption in security.
2. Demand Transparency and Accountability from AI Vendors: Defenders must become more assertive in questioning AI vendors. Lodge urges security professionals to press vendors on their cost models (e.g., "tokens"), request detailed logging, auditing, and recording capabilities, and demand transparency regarding the benchmarks, safeguards, and internal workings of their AI models. If a vendor cannot provide this, organizations should seek solutions that allow for customization and local deployment. This critical scrutiny ensures that AI tools truly enhance security rather than introducing new blind spots or unmanageable costs.
3. Treat AI as a New Technology, Not a Wholly New Threat Vector: Lodge advises treating AI vulnerabilities, such as prompt injection, as analogous to existing attack vectors like SQL injection. This perspective allows defenders to leverage established security principles and safeguards. Implementing robust input validation, output sanitization, access controls (like Attribute-Based Access Control - ARBAC for retrieved documents), and monitoring mechanisms are crucial. Watermarking retrieved documents can also help track data usage and prevent unauthorized dissemination.
4. Embrace GRC as an Engineering Discipline for Continuous Compliance: The talk advocates for transforming GRC from a reactive, audit-driven function into a proactive, continuous engineering discipline. Defenders should leverage AI to perform real-time gap analysis by continuously scanning for new regulations and vulnerabilities, then automatically mapping them against internal policies and controls. This enables organizations to identify and remediate compliance gaps much faster than traditional annual audit cycles. Tools like Audit Caddy demonstrate how this can be achieved by upserting policy documents into vector databases and running classification models against them.
5. Democratize Tools and Consolidate Workflows: AI has the potential to simplify complex security operations by synthesizing information and consolidating the multitude of tools and applications security analysts currently use. Defenders should explore how AI can act as an intermediary to streamline investigations (e.g., for DDoS alerts), reducing the cognitive load and enabling quicker, more informed decision-making.
6. Invest in Education and Upskilling: The successful integration of AI requires knowledgeable personnel. Organizations need to invest in training project managers, security teams, and the next generation of cybersecurity experts to understand AI's architecture, cost implications, and deployment considerations. This includes fostering deeper technical understanding beyond superficial knowledge of AI's capabilities.
7. Start Small and Iterate: Lodge advises organizations to "start small" with AI implementations. RAG, with its controlled environment and focus on proprietary data, is an excellent entry point. By integrating AI incrementally and iterating on solutions, organizations can build confidence, refine their processes, and scale their AI capabilities securely.

By adopting these defensive implications, organizations can harness the power of AI to strengthen their security posture, improve compliance, and gain a competitive edge in a rapidly evolving threat landscape, all while maintaining control and transparency.

Key Takeaways

• RAG is a Secure and Cost-Effective AI Foundation: Retrieval Augmented Generation (RAG) is presented as the premier AI technique for cybersecurity and GRC, offering superior security, privacy, and cost control by allowing organizations to use their own, latest data on-prem or within a VPC, significantly mitigating hallucination risks and intellectual property exposure.
• AI Transforms GRC into a Proactive Discipline: AI can revolutionize GRC by automating gap analysis, streamlining policy generation, and enabling continuous compliance mapping against frameworks like ISO 27001, NIST CSF, and SOC 2, moving organizations from reactive annual audits to proactive, real-time risk management.
• Transparency and Accountability are Non-Negotiable: Organizations must demand complete transparency from AI vendors regarding cost models (e.g., "tokens"), backend workings, logging, auditing, and safeguards. Without this, AI adoption risks creating new layers of opacity and unmanageable expenses.
• Human-in-the-Loop is Essential for Responsible AI: AI functions best as an "information ally" and guide, not a fully autonomous guard. Human oversight, validation, and ethical decision-making remain indispensable to ensure accuracy, prevent misuse, and address the inherent limitations of AI models.
• Self-Hosting Offers Control and Value: Deploying AI solutions like RAG in a self-hosted environment (e.g., a dedicated GPU in a VPC) provides greater control over data, enhanced security, and can be more cost-effective in the long run compared to relying solely on third-party cloud AI APIs, especially when considering the implicit cost of data privacy.
• Open-Source Tools Empower Practical Implementation: Projects like Audit Caddy demonstrate the practical application of AI (using BERT models and vector databases) for automating GRC tasks such as policy classification, search, generation, and compliance reporting, making advanced AI capabilities accessible to a broader audience.

About the Speaker(s)

Brennan Lodge is a seasoned expert in financial services and cybersecurity, bringing 15 to 18 years of experience from prominent institutions such as JP Morgan, the Federal Reserve, Bank of New York, Bloomberg, and Goldman Sachs. He began his career on the blue team, working as a SOC analyst, providing him with a strong defensive cybersecurity foundation.

Lodge holds a Master's degree in Data Science from NYU, obtained during a period when the field was evolving from "big data" and "prediction" to the current focus on AI. He currently serves as a professor and researcher at a university, actively involved in developing and testing AI solutions for real-world problems. Additionally, he works as a fractional CISO, parachuting into companies to help them navigate complex security and compliance challenges.

A vocal advocate for open-source initiatives, Lodge is the creator of Audit Caddy, an open-source tool designed to automate GRC processes using AI. He is also an instructor for LinkedIn Learning, where he teaches courses on RAG (Retrieval Augmented Generation) and MCP (Machine Comprehension Protocol), focusing on realistic, open-source use cases. His expertise spans the practical application of AI in enhancing cybersecurity defenses and streamlining compliance, with a strong emphasis on transparency, cost-efficiency, and data privacy.