One Search To Rule Them All: Threat Modelling AI Search

One Search to Rule Them All: Threat Modelling AI Search

Speakers: Kane Narraway

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch on YouTube

Reading time: ~7 minutes

TL;DR

Enterprise AI search tools like Glean consolidate access to every connected data source behind a single query interface — and that consolidation is precisely what makes them a high-value security target. Kane Narraway from Canva walked through the threat model for AI search, demonstrated real-world authorization issues discovered during Canva's Glean rollout, and drew a direct line from enterprise search to the emerging MCP ecosystem, where the same risk patterns repeat.

Introduction

Most AI security discussions focus on building AI platforms — securing Claude, OpenAI, and other LLM providers — with heavy emphasis on the OWASP LLM Top 10. That guidance is useful for teams building AI infrastructure. It is less useful for the far larger group of security practitioners tasked with evaluating AI-powered SaaS tools their organizations are adopting.

Narraway, a security engineer at Canva based in New Zealand, set out to fill that gap. His target: enterprise AI search. Tools like Glean, Guru, Atlassian Rovo, and a growing list of competitors promise to do for internal knowledge what Google does for the web — one interface to find anything across Confluence, Slack, Google Drive, Jira, and every other connected application. The pitch is compelling. The security surface area is significant.

What Enterprise AI Search Actually Does

▶ Watch: How AI search works under the hood (04:00)

The user-facing experience is simple: type a question, get a natural language answer. What happens beneath the surface is more complex. When a query arrives, the tool makes a series of API calls to every connected data source, indexes the results in a vector database, passes the indexed content to an LLM provider of the administrator's choosing, and returns a summarized response — all in the time it takes to display a result.

From a security standpoint, this architecture condenses risk. An attacker who previously needed to compromise three separate tools now needs to compromise one session cookie. And the number of these tools is accelerating: Narraway noted that the slide he made listing AI search vendors was already out of date by the time he presented it.

At Canva, the security and IT teams chose to roll out Glean. Narraway spent time examining it closely, as well as reviewing Guru and spending time with Atlassian Rovo — making his findings broadly applicable across the category.

Threat Model: Integrations, Access Control, and Zero Trust

▶ Watch: Threat model framework for AI search (07:30)

Narraway organized the threat model for AI search into four categories: integration risk, third-party and privacy risk, access and authorization risk, and standard SaaS risks. He skipped the latter two as largely equivalent to any other SaaS deployment, focusing instead on integrations and access control as the most distinctive and underexamined areas.

Integration risk starts with a deceptively simple question: which data sources should the tool be allowed to connect to? Slack is a useful example of why this question is harder than it appears. Slack has a layered structure — grids, workspaces, public channels, private channels, and direct messages. Connecting any of these to an indexing tool is technically possible. Whether it is appropriate depends on the organization's norms. Narraway noted that many employees are uncomfortable having their private DMs indexed, and that DEI channels or sensitive internal communications surfacing through a search interface can create real organizational issues.

Even for public channels, the act of connecting increases discoverability risk: content that was technically accessible but practically obscured by the friction of knowing where to look becomes trivially findable. Narraway recommended doing keyword searches across connected content at rollout to surface anything that may have been inadvertently exposed for years.

A more subtle integration problem involves authorization architecture built on top of third-party APIs. Narraway illustrated this with Atlassian's Confluence. Ecosystem apps in Atlassian are installed instance-wide and cannot be locked down to a specific space or user. Glean's search effectively checks access permissions at query time — but building reliable authorization on top of APIs that change, have quirks, and occasionally behave unexpectedly is genuinely difficult. In testing, the Canva team found that archiving a Confluence page, moving it, deleting it, and then restoring it wiped the access permissions — making the restored page accessible to Glean in cases where it should not have been. Not a Glean problem per se, but a consequence of building authorization on top of a third-party API.

The more serious finding: while running Glean in a self-hosted configuration (rather than the cloud version), the team assumed their API keys were safely contained within their own infrastructure. They were not. Atlassian's API had been placed behind a proxy that Glean connected to their infrastructure — meaning the API key had external exposure the team was unaware of. The issue has since been fixed in coordination with Glean, but it illustrates the class of surprising trust relationships created when one service acts as a connector between others.

Zero trust breakage is another category. If Confluence is protected by a managed-device policy while Slack is accessible from a personal phone, and Glean is rolled out on mobile devices while connected to Confluence, the carefully constructed device-posture requirement for Confluence has effectively been bypassed via Glean. Narraway's recommendation: place the AI search tool in the highest security tier of the organization's zero trust policy and restrict access to managed devices.

What AI Search Can Teach You About MCP

▶ Watch: MCP parallels and securing agentic access (22:00)

Narraway used the second half of his talk to draw the connection between enterprise search and the Model Context Protocol — a bridge that illuminates both.

MCP, in Narraway's framing, is essentially a layer in front of organizational APIs that translates LLM prompts into specific actions. Where Glean centralizes search, MCP centralizes agentic capability. The security threat model is nearly identical: integrations, access control, service accounts, where the interface is deployed, which tier of zero trust it sits in.

The current deployment reality for MCP is messy. Many engineers are downloading MCP server packages from package managers or GitHub and running them locally on their workstations — functionally no different from downloading random software from the internet. Canva's "Otter" agent is an example of the alternative: a centrally hosted MCP server that the security team has reviewed, that uses standardized access controls, and that employees interact with through a single interface rather than a proliferation of locally installed packages.

Cloudflare and others have begun offering server-side MCP hosting — an architecture Narraway recommended. Centralized hosting enables proper logging and monitoring, eliminates the package manager risk of multiple unvetted versions, and lets organizations enforce one well-understood version of each integration. The tradeoff is the loss of local file system access, which some MCP use cases require.

His recommended MCP security process mirrors the enterprise search approach: audit what MCP servers employees are already using (using EDR, MDM, or OSQuery), security-review those, deploy approved ones via workstation management tools like Jamf or Intune, and provide a "paved path" that makes the secure option the easiest option.

Three Principles That Apply Across Both

▶ Watch: Summary and takeaways (27:00)

Narraway closed with three principles he argued generalize across all AI SaaS tools, not just enterprise search:

1. Every app is now an AI app. When threat modeling any SaaS tool, the question is no longer "is this an AI app?" — it is "what is it integrating to, and where is it being accessed from?" Those two questions cover the majority of the unique risk.

1. Connectivity increases risk surface area. This is not a reason to avoid AI tools, but it must be accepted as a constant. Good access controls, data classification, and integration reviews help but do not eliminate the risk. The risk exists even after doing all of those things well.

1. Secure service accounts. All of this infrastructure is powered by non-human identities. Service accounts that use what is functionally single-factor authentication — a token that, if exposed to the internet, can be used without further challenge — are the operational backbone of enterprise AI search and MCP. IP allowlisting, minimal scope, and strong monitoring of service account usage are not optional hygiene items.

Notable Quotes

"Rather than needing to compromise three tools, now I only need to compromise the session cookie for one." — Kane Narraway (04:30)

"Building auth is hard. Building auth on top of auth is even harder. Building auth on top of auth when the APIs change every day is probably one of the hardest things you can do." — Kane Narraway (14:00)

"Most people aren't going to do them unless you've got a lot of time and a lot of high fidelity signals that you can alert on." — Kane Narraway, on tactical controls (20:00)

Key Takeaways

• Enterprise AI search collapses the attack surface. Compromise one session token and access everything connected to the tool — a significantly different risk profile than compromising individual applications separately.
• Authorization built on top of third-party APIs is fragile. Quirks in upstream APIs (like Atlassian's permission behavior after page restoration) can cause AI search tools to surface documents that should be restricted.
• Zero trust configurations can be silently bypassed. Connecting AI search to a protected resource while making that search available on mobile devices effectively voids managed-device requirements for that resource.
• MCP shares the same threat model as enterprise search. Integrations, access control, where the interface runs, and service account hygiene are the dominant risk categories for both.
• Centralized, security-reviewed MCP deployment is the right path. Auditing what MCP servers employees are already running, reviewing them, and deploying via workstation management dramatically reduces supply chain and operational risk.