Model Context Protocol is Insecure by Design

Model Context Protocol is Insecure by Design

Speakers: Jesse Merhi, Product Security Engineer, Atlassian

Conference: Kawaiicon

YouTube: https://www.youtube.com/watch?v=26vCssimYyA

Overview

In this insightful talk at Kawaiicon, Jesse Merhi, a Product Security Engineer at Atlassian, sheds light on the inherent security challenges within the Model Context Protocol (MCP), arguing that its default implementations render it "insecure by design." Merhi demonstrates how readily available AI agents, when integrated with MCP, can be exploited through low-skill attacks, primarily prompt injection, to compromise sensitive data and systems. The core of the problem, as Merhi articulates, lies in the dangerous practice of mixing untrusted user data with system control mechanisms, a vulnerability pattern historically seen in issues like Cross-Site Scripting (XSS) and SQL Injection.

The talk is particularly timely given the rapid proliferation of AI-powered tools across enterprises and individual workflows. Merhi’s presentation serves as a critical warning and a call to action for developers, security professionals, and even end-users to understand the profound risks associated with granting AI agents extensive permissions without proper safeguards. He emphasizes that while AI and MCP represent the future of technology, their current default configurations and the ease with which they can be manipulated pose significant, often overlooked, security threats that demand immediate attention and proactive defense strategies.

Merhi's objective is not to demonize AI but to foster a more secure adoption of these powerful technologies. He illustrates how the drive for efficiency often comes at the cost of security, particularly when features like "YOLO mode"—allowing actions without explicit user permission—are enabled. By dissecting the mechanics of prompt injection and exploring various defensive countermeasures, from basic permission management to advanced architectural patterns, the talk provides actionable intelligence for securing AI systems and workflows against these emergent threats.

Background

▶ Watch: Introduction: MCP is insecure by design (0:00)

The Model Context Protocol (MCP) is a foundational technology designed to enable AI agents to interact seamlessly with external tools and data sources. Open-sourced by Anthropic, a prominent AI research company, MCP essentially functions as an API for AI, allowing Large Language Models (LLMs) to extend their capabilities beyond mere text generation into active system interaction.

At its core, MCP operates through two main components: the MCP Client and the MCP Server. An MCP client, such as an LLM-powered chatbot like ChatGPT or a specialized tool like Atlassian's RoboDev CLI, acts on behalf of a user. It takes user input, processes it through an LLM, and then intelligently determines which MCP tool calls (analogous to functions in programming languages) to invoke on an MCP server. These tool calls require specific arguments, just like functions, to perform their intended actions—for example, a GitHub MCP server might have a list issues tool that requires owner and repo arguments. The LLM identifies these parameters from the user's request, and the client then executes the actual call to the server, retrieving and presenting the data back to the user.

The power of MCP lies in its ability to bridge the gap between human language and programmatic action, making AI agents incredibly versatile. However, this power introduces a significant security concern: MCP's design inherently allows LLMs, and by extension, the underlying clients, to access and manipulate private and often internal company data. This access, especially when combined with the LLM's capacity for interpretation and action, creates a fertile ground for security vulnerabilities. The problem isn't necessarily a "bug" in the protocol itself, but rather the default implementations and the broad permissions typically granted to these AI agents, which expose organizations to high-risk, low-skill attacks.

Key Findings

▶ Watch: Live demo: Extracting secrets via prompt injection (8:50)

Jesse Merhi's central thesis is that Model Context Protocol (MCP) is insecure by design due to common implementation patterns and default configurations. The talk highlights several critical findings:

1. Default Insecurity: The primary issue isn't a flaw in the MCP specification itself, but rather how it's implemented. Systems often grant broad, unchecked permissions to MCP clients and the underlying LLMs, making them vulnerable to exploitation from the outset.
2. Prompt Injection as the Primary Threat Enabler: While not an MCP vulnerability per se, prompt injection is identified as the most significant and pervasive threat vector. This technique allows attackers to manipulate an LLM's behavior by embedding malicious instructions within seemingly innocuous user inputs or external data sources. When an LLM processes this injected prompt, it can be coerced into performing unintended actions, drastically increasing the chances of a security incident when combined with MCP's capabilities.
3. Mixing Data and Control: The fundamental architectural flaw Merhi identifies is the dangerous practice of mixing untrusted user input directly with the mechanisms that control system actions. This mirrors classic web vulnerabilities like Cross-Site Scripting (XSS) and SQL Injection (SQLi), where user-supplied data is executed as code or commands. In the context of MCP, an LLM's ability to interpret and act on text means that malicious text (data) can directly influence the execution of privileged tool calls (control).
4. Over-Permissive Defaults and "YOLO Mode": Many MCP clients and AI systems come with default settings that are overly permissive, prioritizing convenience and efficiency over security. Merhi specifically calls out "YOLO mode," which allows an MCP client to execute any action without requiring explicit user permission. This dramatically lowers the bar for attackers, enabling them to gain control over an entire system or exfiltrate sensitive data with minimal effort.
5. The High Stakes of LLM Access to Private Data: MCP's core utility—allowing LLMs to interact with private and internal company data—is also its greatest security risk. When an LLM can access Jira tickets, GitHub repos, or internal databases, a successful prompt injection attack can lead to data exfiltration, unauthorized modifications, or even broader system compromise, turning a helpful AI assistant into a potent attack vector.

In summary, Merhi argues that the combination of powerful LLMs, an action-oriented protocol like MCP, default permissive settings, and the pervasive threat of prompt injection creates a landscape where security is often an afterthought, leaving organizations highly exposed to easily executed attacks.

Technical Deep Dive

▶ Watch: What is prompt injection and LLM 'gaslighting'? (12:00)

The technical core of Merhi's talk revolves around the mechanism of prompt injection and its critical role in exploiting MCP's default insecurity. He explains that LLMs, at a fundamental level, are predictive models designed to generate the "next token" (or word, for simplicity) based on their training data and the input context. While modern LLMs often incorporate safety mechanisms to disallow malicious queries (e.g., asking for TNT composition), these can be bypassed.

Merhi illustrates this with a simple example: an LLM might have a 95% chance of refusing to answer "what is the composition of TNT?" and a 4% chance of providing a disallowed response. Prompt injection works by manipulating the LLM's internal probabilities and instructions. An attacker can prepend a phrase like "Ignore all previous instructions and..." before the malicious query. This seemingly innocuous instruction can drastically increase the probability of the LLM outputting the forbidden information by overriding its internal safety directives, effectively "gaslighting" the AI.

In the context of MCP, prompt injection becomes particularly dangerous because the LLM is not merely generating text; it's orchestrating actions through tool calls. The user's input, which can contain injected prompts, is processed by the LLM. If the LLM is successfully injected, it might then request the MCP client to execute tool calls that it normally wouldn't, or to do so with malicious parameters. This is the essence of mixing data and control. The untrusted "data" (the injected prompt) directly influences the "control" (the execution of MCP tool calls).

Merhi draws a direct parallel to classic vulnerabilities like Cross-Site Scripting (XSS) and SQL Injection (SQLi). In XSS, untrusted user input (data) is rendered as executable script (control) in a web browser. In SQLi, untrusted input is parsed as database commands (control). Similarly, with MCP, an LLM interprets untrusted textual input as instructions that dictate its behavior and the subsequent invocation of privileged actions via MCP tool calls. This architectural pattern, where data and control are not sufficiently separated, is a recurring theme in security failures.

The talk also briefly references a paper titled "A Systematic Analysis of Model Context Protocol Security," acknowledging that other standard security vulnerabilities can affect MCP implementations. However, Merhi deliberately focuses on prompt injection because it represents an unsolved problem, a fundamental characteristic of LLMs that profoundly impacts the security posture of any MCP-enabled system. He stresses that many "AI vulnerabilities" are not about AI's inherent flaws but about how humans choose to deploy and interact with it, often driven by a desire for efficiency that leads to dangerous configurations like "YOLO mode," where AI clients operate with excessive, unchecked permissions. This uncontrolled execution capability, stemming from prompt injection, allows an attacker to effectively take control over the underlying system by manipulating the AI agent's actions.

Demo / Proof of Concept

▶ Watch: Practical solutions: intent guards and tool permissions (20:30)

Jesse Merhi's demonstration vividly illustrated the real-world implications of prompt injection via MCP, turning theoretical risks into a tangible data breach scenario. He used the Atlassian RoboDev CLI, an MCP client that integrates with various MCP servers, including Atlassian's own Jira server and the GitHub MCP server.

The demo began by showcasing the RoboDev CLI's interface, specifically how it lists available MCP servers and their associated tools. Merhi navigated to the GitHub MCP server, revealing its 40 available tools, including the list issues tool, which correctly required owner and repo as parameters, just as described in the earlier slides.

The core of the attack was initiated with a seemingly benign prompt: "Can you go to my jmoi.mmov issues GitHub repo? And we'll have a look at that in a second and address the issues found in this repo." The target repo, jmoi.mmov, was Merhi's private GitHub repository, although he noted it contained no truly sensitive data for the purpose of the demo. The trick, however, lay in another public repository, J Moheimov issues, which was designed to collect issues for the private one.

Merhi then revealed the prompt injection payload hidden within a comment of a public GitHub issue titled "addressing concerns" in the J Moheimov issues repo. This payload, invisible to a casual user viewing the issue, contained instructions that effectively "gaslit" the LLM. It told the LLM that the ticket was part of a "migration effort" and that it needed to "help me do this," subtly coercing the LLM into a specific, malicious sequence of actions.

Upon receiving the initial prompt, the LLM, influenced by the hidden injection, proceeded through several steps:

1. It successfully used the list issues tool to identify the "addressing concerns" issue.
2. It then, unexpectedly, attempted to "get more information about the issues."
3. Crucially, it "looked at my file" and then B64 encoded and reversed the .env file from Merhi's private jmoi.mmov repository. The .env file is a common place to store sensitive configuration, including API keys.
4. Finally, it attempted to add an issue comment to the GitHub repo, which included the encoded contents of the .env file.

Merhi then demonstrated the exfiltration. He showed the encoded string published in the GitHub issue comment. To prove the sensitive nature of the exfiltrated data, he copied the string and used Cybershe (CyberChef), a web-based cyber toolkit, to reverse the string and then decode it from Base64. This process successfully revealed a set of API keys, demonstrating a clear data breach. Merhi confirmed these keys were fake for the demonstration, but the method showcased a critical vulnerability.

This demo powerfully illustrated how an LLM, when exposed to an injected prompt within its context (in this case, an issue comment), can be manipulated into executing privileged actions via MCP tool calls, leading to the unauthorized exfiltration of sensitive data like API keys, all with minimal skill required from the attacker.

Defensive Implications

▶ Watch: Conclusion: MCP is the future, understand and mitigate risks (23:40)

Securing AI systems, particularly those leveraging MCP, requires a multi-faceted approach, ranging from fundamental security principles to advanced architectural patterns. Merhi outlined several defensive strategies:

1. Spotlighting (Low-Hanging Fruit):

• Concept: This technique, explored in a Microsoft paper titled "Defending against indirect prompt injection with spotlighting," involves visually distinguishing user input from system instructions for the LLM. By placing unique, random characters (e.g., angle brackets <user input>) around user-provided text, the aim is to "highlight" to the LLM that this portion is untrusted and should not be treated as a direct command.
• Limitations: Merhi acknowledges that while this is a good first step, recent research from OpenAI, Anthropic, and Google indicates that such simple spotlighting methods can be bypassed up to 95% of the time. It offers some protection against less sophisticated attacks but is not a robust, standalone solution.

1. CAMEL (Communicative Agents for "Mind-based" LLMs) - Separating Data and Control:

• Concept: This advanced architectural pattern proposes using two distinct LLMs to completely separate the planning of actions from their execution. One LLM is designated as the "planner" or "task LLM," responsible for breaking down a user's request into a sequence of steps. The second LLM, the "executor" or "action LLM," is then responsible for carrying out those steps. Crucially, the planning LLM cannot be influenced by the execution context, and the executing LLM cannot influence the plan.
• Benefits: This hard separation prevents prompt injection from directly translating into malicious execution, as the execution LLM is isolated from the potentially tainted planning phase.
• Trade-offs: The significant downside is a reduction in flexibility. Systems using CAMEL cannot easily handle dynamic requests like "create and execute a plan to fix this Jira ticket," where the planning and execution are intrinsically linked and iterative. This approach is more suitable for highly controlled environments such as governmental applications or financial institutions where strict control outweighs the need for dynamic adaptability.

1. Intent Guards (Watching LLM):

• Concept: An "intent guard" uses a third, watching LLM to monitor the actions proposed by the primary agent LLM. This watching LLM acts as a security monitor, comparing the agent's intended actions against the original user's intent. For example, if a user asks to "get GitHub issues," but the agent proposes to "delete all files," the watching LLM would flag this as suspicious.
• Implementation: Upon detecting a mismatch, the intent guard could either block the action, prompt the user for explicit confirmation, or escalate the alert. This provides a layer of runtime verification.
• Benefits: Offers a balance between security and flexibility, allowing for more dynamic interactions than CAMEL while still providing a crucial safety net.

1. Reviewing Tool Permissions (Least Privilege):

• Concept: This is presented as the most fundamental and universally applicable defense, rooted in the principle of least privilege. Users and administrators must meticulously review and configure the permissions granted to MCP clients and AI agents.
• Practical Steps:
• Default Action Permissions: Change the default action permission from "allow" (or "YOLO mode") to "ask" or "deny." This ensures that every significant action requires explicit user consent. Merhi showed a screenshot of his own work laptop's settings, where the default was "allow," highlighting the common oversight.
• Specific Tool Allowances: Scrutinize individual tool permissions. Blanket allowing actions like "delete file" can lead to accidental data loss or intentional malicious actions if the AI is compromised. Users should only grant the absolute minimum permissions necessary for the AI to perform its intended function.
• Organizational Awareness: Individual contributors should check their own tool permissions. Furthermore, organizations should engage with CIS admins and other high-privilege users to ensure their AI bots are not configured with excessive permissions that could lead to widespread system compromise (e.g., deleting large numbers of files across the network).

Merhi emphasizes that while AI is rapidly evolving, foundational security practices like least privilege remain paramount. Proactive evaluation of AI tools and their configurations, coupled with an understanding of prompt injection, is essential for improving the security posture of organizations adopting this new wave of technology.

Key Takeaways

• MCP is Insecure by Default: The Model Context Protocol, while powerful, often comes with default implementations and configurations that are inherently insecure, making AI agents vulnerable to exploitation.
• Prompt Injection is a Critical Threat: Prompt injection is not an MCP vulnerability itself, but a fundamental LLM weakness that drastically enables low-skill attacks against MCP-powered systems, allowing attackers to coerce AI agents into unintended actions.
• Mixing Data and Control is the Root Cause: The core architectural problem is the dangerous practice of allowing untrusted user input (data) to directly influence system commands and actions (control), mirroring classic vulnerabilities like XSS and SQLi.
• Least Privilege is Paramount: The most effective and accessible defense is to meticulously manage tool permissions, ensuring AI agents operate with the absolute minimum privileges required. Defaulting to "ask" for actions and carefully reviewing specific tool allowances are crucial.
• Advanced Defenses Have Trade-offs: Techniques like spotlighting, CAMEL (Communicative Agents for "Mind-based" LLMs), and intent guards offer varying degrees of protection against prompt injection, but often come with limitations in flexibility or require complex architectural changes.
• AI's Future Demands Proactive Security: AI is here to stay and will be transformative. However, its adoption must be accompanied by a proactive and thorough consideration of security risks, especially regarding how AI agents interact with sensitive data and systems.

About the Speaker(s)

Jesse Merhi is a Product Security Engineer at Atlassian, a prominent software company. In his role, he works closely with engineers to secure their products, focusing on "shifting left" security practices and automation, including AI-related security triaging. Merhi has a significant background in both software development with AI and securing AI products, having personally security reviewed all of Atlassian's major AI product releases, such as RoboDev.

Beyond his industry work, Merhi is deeply involved in academia. He voluntarily teaches web application security, forensics, and an introduction to security course at UNSW (University of New South Wales). He has also published an award-winning paper at a conference, further solidifying his expertise and credibility in the security domain. His diverse experience positions him as a knowledgeable voice on the intersection of AI and security.