RAGe Against the Machine: AI-Powered Compliance and Cybersecurity
Brennan Lodge
Overview
In his CypherCon talk, "RAGe Against the Machine: AI-Powered Compliance and Cybersecurity," Brennan Lodge, co-founder and CISO at the Manhattan Institute, delves into the transformative potential of Artificial Intelligence (AI) in navigating the complex landscapes of cybersecurity and regulatory compliance. The presentation dissects the current state of AI adoption, exposing both its promises and perils, while championing a strategic, data-driven approach to integration. Lodge introduces two pivotal frameworks: Retrieval Augmented Generation (RAG) and the Model Context Protocol (MCP), demonstrating how these technologies can provide tangible, auditable, and cost-effective solutions to pressing industry challenges.

Key moments
- 5:30 AI's benefits, pitfalls, and challenges in cybersecurity
- 17:48 Explaining Retrieval Augmented Generation (RAG) with a library analogy
- 20:38 Introducing Model Context Protocol (MCP) for LLM communication
- 22:20 Practical GRC use cases using RAG and MCP for compliance
- 26:30 AI for policy gap analysis and compliance framework translation
- 28:50 Open-source compliance tools and Audit Caddyy product
- 34:00 Q&A on scaling AI and the human-in-the-loop
RAGe Against the Machine: AI-Powered Compliance and Cybersecurity
Speakers: Brennan Lodge
Conference: CypherCon
YouTube: https://www.youtube.com/watch?v=4UzcaMZmBwA
Overview
In his CypherCon talk, "RAGe Against the Machine: AI-Powered Compliance and Cybersecurity," Brennan Lodge, co-founder and CISO at the Manhattan Institute, delves into the transformative potential of Artificial Intelligence (AI) in navigating the complex landscapes of cybersecurity and regulatory compliance. The presentation dissects the current state of AI adoption, exposing both its promises and perils, while championing a strategic, data-driven approach to integration. Lodge introduces two pivotal frameworks: Retrieval Augmented Generation (RAG) and the Model Context Protocol (MCP), demonstrating how these technologies can provide tangible, auditable, and cost-effective solutions to pressing industry challenges.
Lodge's talk is particularly relevant in an era defined by information overload, a persistent cybersecurity talent gap, and the ubiquitous, often uncritical, "AI-washing" of products. He argues against the prevailing notion that AI will simply replace junior analysts, instead advocating for its role as a powerful "co-pilot" that empowers human practitioners, enriches context, and accelerates decision-making. By focusing on practical use cases, particularly within Governance, Risk, and Compliance (GRC), Lodge illustrates how AI, when properly architected and governed, can move beyond hype to deliver measurable improvements in security posture and operational efficiency.
The presentation serves as a clarion call for cybersecurity professionals to not only understand the technical underpinnings of AI but also to actively shape its ethical and practical deployment. Lodge challenges attendees to "roll their own" AI solutions, leveraging open-source initiatives to customize AI for specific organizational needs, rather than passively accepting vendor offerings. His insights provide a pragmatic roadmap for integrating AI responsibly, ensuring it acts as a guide rather than an autonomous guard, thereby enhancing security without compromising human oversight or introducing new, unmanageable risks.
Background
▶ Watch: AI's benefits, pitfalls, and challenges in cybersecurity (5:30)
Brennan Lodge's journey into the intersection of AI and cybersecurity began from a foundation in the blue team, working at financial giants like JP Morgan, the Federal Reserve Bank in New York, Bloomberg, Goldman Sachs, and HSBC. His early career highlighted the critical need for better data utilization in security operations. Recognizing the power of data, Lodge pursued a master's degree in data science and analytics at NYU in 2015, a period when terms like data science, machine learning, and big data were emerging. He recalls the earlier promises of User and Entity Behavior Analytics (UEBA), noting how AI has since absorbed and redefined many of those aspirations, becoming the overarching term for advanced analytics in security.
At Goldman Sachs, Lodge established the first data science team dedicated to cybersecurity, developing machine learning algorithms to detect suspicious activity, enhance vulnerability management, and ultimately optimize GRC processes. This experience solidified his belief in data's capacity to objectively tune security controls and improve operational efficiency, moving beyond anecdotal evidence to data-driven justification for security decisions.
However, Lodge critically examines the current state of AI adoption, highlighting "the good, the bad, and the ugly." The "good" stems from the undeniable need for assistance in cybersecurity: combating information overload, addressing the 24/7 "follow the sun" model for blue teams, and bridging the significant talent gap contributing to analyst burnout. He emphasizes AI's potential to empower junior analysts, providing crucial context enrichment for Security Information and Event Management (SIEM) alerts and accelerating their understanding of complex network topologies and vulnerabilities.
The "bad" manifests in what Lodge dubs the "Salt Bae" phenomenon—an indiscriminate sprinkling of AI onto every product without clear purpose or demonstrable Return on Investment (ROI). This uncritical adoption leads to shadow AI (unauthorized AI tools appearing in corporate environments, raising data confidentiality concerns) and significant, often unforeseen, cloud costs associated with GPU usage and AI model token consumption. Lodge warns against the pitfalls of treating AI as merely a glorified version of "Clippy," the annoying Microsoft Office assistant, underscoring the need for meaningful integration.
CISOs, according to Lodge, face a triple dilemma: ensuring AI solutions are secure (with traceability and auditability), timely (without sacrificing human oversight for speed), and cost-effective. He advocates for AI as a "guide," not a "guard," meaning it should augment human capabilities rather than autonomously making critical decisions, especially those that could violate security controls (e.g., SSH as root). The background sets the stage for introducing RAG and MCP as frameworks designed to address these challenges by providing controlled, auditable, and intelligent AI assistance.
Key Findings
▶ Watch: Introducing Model Context Protocol (MCP) for LLM communication (20:38)
Brennan Lodge's talk reveals several key findings and contributions to the field of AI-powered cybersecurity and compliance. Central to his presentation are the introductions and advocacy for Retrieval Augmented Generation (RAG) and the Model Context Protocol (MCP) as foundational architectural components for secure and effective AI integration.
Firstly, Lodge identifies RAG as a powerful framework to overcome the inherent limitations of traditional Large Language Models (LLMs). LLMs, being static binaries, struggle with integrating new, real-time, or proprietary knowledge and are prone to "hallucinations" (generating factually incorrect but plausible-sounding information). RAG addresses this by augmenting the LLM with external knowledge bases, specifically a vector database containing an organization's internal data (policies, threat reports, alert inventories). This allows the LLM to provide responses grounded in verifiable, specific information, thereby increasing trustworthiness and reducing the time lag associated with traditional LLM updates. Lodge's experiments demonstrated RAG could achieve retrieval times under 10 seconds and operational costs under $500 per month, making it a practical solution for enterprise environments.
Secondly, Lodge introduces the Model Context Protocol (MCP), conceptualizing it as a standardized, governed connector for LLMs. Likening it to a universal USB-C port, MCP provides a single, controlled interface for an AI model to communicate with diverse external or remote services, such as SIEM queries, regulatory databases, or threat intelligence platforms. A critical finding is MCP's robust security feature: it enables comprehensive logging of every interaction, recording who made the call, which AI model was used, timestamps, the AI's reaction, and even the top relevant text choices from the vector database. This provides crucial traceability and auditability, addressing a primary concern for CISOs regarding AI governance.
Thirdly, the talk highlights the immense potential of AI, specifically through RAG and MCP, to revolutionize Governance, Risk, and Compliance (GRC). Lodge demonstrates how AI can automate the arduous task of interpreting complex, multi-page regulatory documents (often "written by politicians") and translating them into actionable security controls. His work shows that AI can effectively map internal policies (e.g., employee handbooks) to specific controls, identifying compliance gaps with a measurable percentage match using models like BERT. This capability drastically reduces the manual effort and expertise required for compliance, offering a "compliance notebook" approach that leverages existing compliance data (e.g., SOC 2, ISO) for new frameworks like ISO 42001 (the AI framework).
Finally, Lodge underscores the importance of open-source contributions in this domain. He mentions his GitHub repository, which hosts BERT models, policy templates, evidence, and frameworks, translating them into compliance notebooks. This open-source approach empowers organizations to "roll their own" AI solutions, fostering innovation and reducing reliance on expensive, proprietary vendor lock-ins. He also alludes to a commercial product, AuditCaddy, which embodies these AI-driven compliance automation principles, further validating the practical applicability of his research.
Technical Deep Dive
▶ Watch: Practical GRC use cases using RAG and MCP for compliance (22:20)
The core of Brennan Lodge's technical presentation revolves around two complementary architectures: Retrieval Augmented Generation (RAG) and the Model Context Protocol (MCP). These frameworks are designed to provide AI with context, control, and auditability, moving beyond the limitations of standalone Large Language Models (LLMs).
Retrieval Augmented Generation (RAG)
Lodge explains RAG using the analogy of a library:
- LLM (Librarian): This is the core language model, responsible for understanding queries and generating responses. However, as Lodge notes, LLMs are "static" binaries; their knowledge base is fixed at the time of training. They cannot inherently access new information or proprietary organizational data without external augmentation.
- Vector Database (Library's Index/Shelves): This is a specialized database optimized for storing and retrieving "embeddings"—numerical representations of text, images, or other data that capture semantic meaning. Unlike traditional relational databases that rely on exact matches or indexed keywords, vector databases can perform similarity searches based on context. Lodge emphasizes their efficiency in understanding context, allowing for powerful queries that go beyond simple keyword lookups. This database houses an organization's specific, proprietary, and up-to-date information, such as internal policies, threat intelligence reports, vulnerability data, or regulatory frameworks.
- Sentence Embedding Model (Translator): This component acts as the bridge between human language and the vector database. When a user queries the RAG system, the sentence embedding model translates the natural language query into a numerical vector (an embedding). This query vector is then used to search the vector database for semantically similar documents or text chunks.
How RAG Works:
- A user submits a query to the RAG system.
- The Sentence Embedding Model converts the query into a vector representation.
- This query vector is used to search the Vector Database for the most relevant documents or passages.
- The retrieved context (e.g., specific policy paragraphs, sections of a regulation) is then provided to the LLM alongside the original user query.
- The LLM uses this retrieved context to generate a more accurate, informed, and verifiable response, effectively reducing hallucinations and grounding the output in factual, organization-specific data.
Key Advantages of RAG:
- Dynamic Knowledge Integration: RAG allows LLMs to access and incorporate new information without requiring expensive and time-consuming retraining of the base model.
- Reduced Hallucinations: By providing explicit context from trusted sources, RAG significantly mitigates the LLM's tendency to generate incorrect information.
- On-Premise Deployment: Organizations can deploy the vector database and even the LLM on-premises or within their private cloud environment (VPC), ensuring data sovereignty and control.
- Contextual Enrichment: RAG enriches responses with relevant, verifiable sources, providing links to the original documents for "trust but verify" in security investigations.
- Cost and Performance: Lodge aimed for RAG implementations costing under $500/month and retrieval times under 10 seconds, demonstrating its practical viability.
Model Context Protocol (MCP)
Lodge introduces MCP as a conceptual "dongle" or a universal connector, akin to a USB-C cable for AI models. MCP defines how an AI model (the client) communicates with external or remote services (the server) in a governed and secure manner.
MCP Architecture Components:
- AI Model (Client): The LLM or other AI application running on a local host (e.g., a laptop).
- MCP Server: A server component, ideally kept on-premises or within a controlled environment, that acts as an intermediary.
- External/Remote Services: These are the diverse data sources and tools that the AI model needs to interact with, such as:
- SIEM systems (for querying alerts).
- Regulatory databases (for compliance information).
- Vulnerability management platforms.
- Threat intelligence feeds.
How MCP Works:
- An AI model (client) sends a request to the MCP Server.
- The MCP Server, acting as a "single governed connector," routes the request to the appropriate External/Remote Service.
- The MCP Server can integrate with a Vector Database (as part of a RAG setup) to fetch relevant data before passing it to the LLM.
- The response from the external service is then relayed back to the AI model via the MCP Server.
Key Security and Governance Benefits of MCP:
- Centralized Control: MCP provides a single, auditable point of access for AI models to interact with sensitive systems, preventing uncontrolled or "shadow AI" access.
- Comprehensive Logging: A paramount feature of MCP is its ability to log every aspect of the transmission:
- Who made the call (user identity).
- What AI model was used.
- Timestamps of the interaction.
- How the AI model reacted (the generated response).
- On the vector database side, what were the top three choices of text relevant to the question, providing insight into the AI's reasoning.
- Traceability and Auditability: This detailed logging ensures a complete chain of thought for AI actions, crucial for regulatory compliance and incident response.
- Security Controls Integration: MCP allows for the insertion of "tolls" or junctures where human intervention or additional security checks can be enforced, preventing the AI from making unauthorized or risky decisions (e.g., "don't SSH into my box as root").
Together, RAG and MCP form a robust technical foundation for deploying AI in cybersecurity and compliance, offering both enhanced intelligence and critical governance capabilities.
Demo / Proof of Concept
▶ Watch: Open-source compliance tools and Audit Caddyy product (28:50)
While Brennan Lodge's talk did not include a live, explicit "demo" in the traditional sense, he effectively presented several compelling use cases that serve as practical proofs of concept for RAG and MCP in real-world scenarios, particularly within Governance, Risk, and Compliance (GRC). These examples illustrate how his architectural vision translates into tangible solutions for complex problems.
One significant proof of concept revolved around a GRC challenge faced at a large financial institution. The head of GRC tasked Lodge with a scenario: "Say we're going to Iceland and we're going to build a bank there. What laws do we need to abide by? Are we covered with the policies that we already have? What are our gaps? And help me interpret that." This highly practical use case demonstrated the power of RAG. By ingesting the bank's internal policies (from Confluence and SharePoint) into a vector database alongside relevant Icelandic and international regulations, the RAG system could query this combined knowledge base. The LLM, augmented by the retrieved context, could then identify specific legal obligations, map them against existing internal policies, and highlight areas where new policies or controls were needed, effectively interpreting complex legal texts and pinpointing compliance gaps. Lodge confirmed that this implementation yielded "really good results," despite initial challenges with context window configuration and data ingestion.
Another powerful proof of concept involved using a BERT model for policy-to-control mapping. Lodge described how this model, known for its sentence completion capabilities, could be configured to understand the semantic gaps between different documents. In this application, internal organizational policies (e.g., an employee handbook) were ingested, along with specific security controls (e.g., from NIST CSF or ISO 27001). The BERT model was then able to determine a "percentage match of context" between a policy statement and a particular control. This capability is exceptionally valuable for bulk uploading policies, evidence, and risks to automatically generate a mapping of how well an organization's documentation aligns with required controls, providing a quantifiable measure of compliance posture.
Lodge also detailed the concept of Compliance Notebooks as a proof of concept for automating the interpretation of new regulations. He highlighted the common challenge of dealing with 100-page PDF regulatory documents, often written in dense, legalistic language. His research, conducted with students (his "lab rats"), revealed that many global frameworks heavily copy from standards like ISO 27001 and NIST CSF. The Compliance Notebooks aim to take a new regulation and "translate it into the controls that we need to abide by." Furthermore, if an organization is already compliant with frameworks like SOC 2 or ISO, the system can leverage that existing compliance data to accelerate adherence to new regulations, such as the upcoming ISO 42001 (the AI framework). This demonstrates how AI can drastically reduce the redundant effort in compliance by intelligently reusing existing evidence and mappings.
To further solidify these proofs of concept, Lodge mentioned his open-source GitHub repository, which provides BERT models, policy templates, and frameworks, translating them into these compliance notebooks. He specifically noted a compliance notebook for CMMC (Cybersecurity Maturity Model Certification) is available, allowing others to "roll their own" AI-powered compliance solutions. He also introduced AuditCaddy, a commercial product that embodies these AI automation principles for compliance, offering a full dashboard for managing frameworks and demonstrating the scalability and commercial viability of his research. These examples collectively serve as robust demonstrations of how RAG and MCP can be applied to solve critical cybersecurity and compliance challenges.
Defensive Implications
▶ Watch: Q&A on scaling AI and the human-in-the-loop (34:00)
The insights presented by Brennan Lodge offer significant defensive implications for cybersecurity practitioners, ranging from empowering junior analysts to fundamentally re-architecting how organizations approach GRC and AI governance.
Firstly, AI, specifically through frameworks like RAG, can serve as a powerful co-pilot for the blue team and Security Operations Center (SOC) analysts. In an environment of information overload and a talent gap, AI can provide critical context enrichment for the endless stream of alerts. A college graduate new to the field, for instance, won't immediately understand an organization's network topology, assets, or specific vulnerabilities. AI can quickly pull relevant information from internal knowledge bases, helping junior analysts make faster, more informed decisions and reducing burnout. This directly counters the narrative that AI will simply replace entry-level positions; instead, it becomes a tool for accelerating their learning and effectiveness.
Secondly, Lodge encourages defenders to use their understanding of AI architectures to "stump the vendor." In a market saturated with "AI-washed" products, knowing the technical details of RAG and MCP allows security leaders to ask probing questions about token costs, auditability, data sovereignty, and the actual mechanisms behind vendor claims. This empowers organizations to make informed purchasing decisions, pushing back against vague promises and demanding concrete ROI.
Thirdly, AI can drive a more objective approach to security control tuning. Lodge's experience building data science teams showed the value of using data to justify turning off outdated or ineffective security rules that consistently generate false positives. AI can analyze alert patterns and historical data to identify such rules, providing the objective evidence needed to optimize SIEM configurations and reduce alert fatigue.
Fourthly, the adoption of AI necessitates robust AI governance. Organizations must develop comprehensive AI policies that dictate how AI is used, who can access it, and what actions it can take. This involves working collaboratively with CTOs and business units to prioritize AI projects and establish clear guardrails. Crucially, the Model Context Protocol (MCP) provides the technical foundation for this governance by enabling traceability and auditability. Defenders can log every AI interaction, understanding the "chain of thought" behind AI-driven decisions, which is vital for regulatory compliance and incident investigation. This ensures AI acts as a "guide" rather than an unchecked "guard" that might violate security controls.
Fifthly, Lodge advocates for fighting fire with fire—using AI to counter increasingly sophisticated attacks. As adversaries leverage AI, defenders must also adopt these advanced capabilities to detect, analyze, and respond to threats more effectively. This necessitates integrating AI into existing security stacks, but critically, doing so with an understanding of the underlying architecture and its security implications.
Finally, the talk emphasizes the indispensable role of the "human in the loop." While AI can provide rapid analysis and context enrichment, the human element remains critical for prioritizing alerts, making final decisions, and providing Reinforcement Learning Human Feedback (RLHF) to continuously improve AI models. Lodge highlights the value of supervisory oversight, where managers can review AI queries made by junior analysts to identify areas of struggle, provide targeted education, and foster a continuous learning environment. This synergistic relationship between human expertise and AI efficiency is paramount for scaling security operations effectively.
Key Takeaways
- RAG and MCP are Foundational for Secure AI: Retrieval Augmented Generation (RAG) provides LLMs with real-time, context-specific, and verifiable information from internal data sources, significantly reducing hallucinations. The Model Context Protocol (MCP) offers a governed, auditable connector for AI models to interact with external services, ensuring traceability and control over AI actions.
- AI Transforms GRC Operations: AI, particularly through RAG-powered systems and BERT models, can automate the complex process of interpreting regulatory documents, mapping internal policies to security controls, identifying compliance gaps, and leveraging existing compliance data for new frameworks, drastically improving efficiency in Governance, Risk, and Compliance (GRC).
- AI as a "Guide," Not a "Guard": AI should augment human capabilities, acting as a "co-pilot" for analysts to provide context enrichment and accelerate decision-making, rather than making autonomous, high-stakes decisions that could introduce new risks or violate security policies without human oversight.
- Governance and Auditability are Paramount: Implementing AI requires robust AI policies and technical frameworks like MCP that enable comprehensive logging, traceability, and auditability of all AI interactions. This ensures accountability, helps manage risks like shadow AI, and allows for human intervention when necessary.
- Challenge Vendors and Manage Costs: Organizations should actively question AI vendors about token costs, ROI, and the technical specifics of their offerings. Understanding the underlying architectures (RAG, MCP) empowers defenders to make informed decisions and manage the often-unforeseen financial implications of AI adoption.
- Embrace Open Source for Customization: Leverage open-source initiatives, like Brennan Lodge's GitHub repository for compliance notebooks, to "roll your own" AI solutions. This fosters innovation, allows for customization to specific organizational needs, and reduces reliance on proprietary vendor lock-ins.
About the Speaker(s)
Brennan Lodge is a distinguished expert at the intersection of cybersecurity and data science, currently serving as the co-founder and CISO at the Manhattan Institute. His extensive career spans prominent roles in the financial sector, including positions at JP Morgan, the Federal Reserve Bank in New York, Bloomberg, Goldman Sachs, and HSBC.
Lodge began his career in the blue team, gaining hands-on experience in security operations. Recognizing the transformative potential of data, he pursued a Master's degree in Data Science and Analytics from NYU in 2015. This academic pursuit equipped him with the "superpowers" of data science, which he then applied to cybersecurity. He is credited with establishing the first data science team at Goldman Sachs dedicated to enhancing cybersecurity, where he developed machine learning algorithms for suspicious activity detection, vulnerability management, and GRC optimization.
A recognized thought leader, Lodge has received awards for his research, particularly in the domain of Retrieval Augmented Generation (RAG). He is also an active educator, serving as a professor at NYU and sharing his expertise through three LinkedIn Learning courses covering RAG, Model Context Protocol (MCP), and AI automation. Lodge is a strong advocate for leveraging AI responsibly to empower cybersecurity professionals, solve complex compliance challenges, and foster innovation through open-source contributions.
Reviews
Dr. Zero (Offensive Security Researcher) — SOLID
Lodge brings actual math to the spreadsheet-heavy hellscape of GRC. Using RAG and the Model Context Protocol (MCP) to automate policy-to-control mapping isn't groundbreaking security research, but it's a highly effective painkiller for the blue team that saves analysts from reading 100-page regulatory PDFs.
Heather Calloway (CISO) — MUST SEE
Lodge strips the vendor hype out of AI and delivers a concrete architectural approach for governing LLMs in the enterprise. By pairing RAG with the Model Context Protocol (MCP), he provides a blueprint for making AI auditable, cost-effective, and strictly useful for GRC. This is required viewing for any security leader trying to move their program past AI-washing and into operational reality.