Mind the Data Voids: Hijacking Copilot Trust

Mind the Data Voids: Hijacking Copilot Trust

Speakers: Tobias Diehl

Conference: DEF CON 33

YouTube: https://www.youtube.com/watch?v=NgjyBKfqJEs

Slides: https://media.defcon.org/DEF%20CON%2033/DEF%20CON%2033%20presentations/Tobias%20Diehl%20-%20Mind%20the%20Data%20Voids%20Hijacking%20Copilot%20Trust%20to%20Deliver%20C2%20Instructions%20with%20Microsoft%20Authority.pdf

Overview

Microsoft Copilot integrates the Bing search engine as a live retrieval backend — when users ask about topics not covered by Copilot's training data, it fetches current information from Bing and uses those results as the basis for its answers. At DEF CON 33, offensive security engineer Tobias Diehl demonstrated that this architecture creates a data void exploitation vulnerability: by publishing attacker-controlled content that Bing indexes and ranks highly, an adversary can inject false information into Copilot's answers and, more critically, use this vector to deliver command-and-control (C2) instructions to implants on target machines — all with the apparent authority of Microsoft's AI assistant.

Diehl framed the attack around a concept he calls data voids — gaps in the Bing search index where authoritative content does not exist, allowing attacker-created content to become the de facto authoritative source. The full title of the talk references "delivering C2 instructions with Microsoft authority" — the ultimate implication is that Copilot can be turned into an unwitting C2 relay that victims and defenders would have difficulty distinguishing from legitimate Microsoft-sourced information.

The research was recognized by Microsoft's bug bounty program, and Diehl was named one of Microsoft's Most Valuable Researchers for the year.

Background

▶ Watch: Architecture: how Copilot retrieves answers from Bing web results (2:25)

Copilot's Retrieval Architecture

Microsoft Copilot (formerly Bing Chat, and still tightly integrated with Bing) operates with a dual knowledge source:

1. Training data: A large pre-trained language model (based on GPT-4 class architecture) with knowledge up to a training cutoff date.
2. Live web retrieval via Bing: For queries about recent events, current prices, policies, or topics not well-represented in training data, Copilot performs a Bing search and uses the top-ranked results as grounding context for its response.

The integration means Copilot inherits Bing's content quality model — it trusts Bing's ranking signals to surface authoritative, accurate content. This trust is not merely a UI preference; it is architecturally embedded in how Copilot composes its answers, including the sources it cites and the tone of confidence it uses.

The Concept of Data Voids

A "data void" (a term from disinformation research) refers to a topic or search query for which little or no authoritative content exists in a search index. Data voids are exploitable because search engines fill the void with whatever is available — and if an attacker publishes content targeting the void, it may rank at or near the top simply by virtue of being the only result.

Examples of natural data voids:

• Very recent events before mainstream media coverage catches up.
• Niche internal company policies not publicly documented.
• Highly technical or obscure procedures that have little existing web documentation.
• Trademarked terms or internal system names that organizations use internally but don't publish externally.

An attacker who identifies a data void relevant to a target — for example, a company's internal security policy name, a specific software configuration keyword, or a recently coined proprietary term — can publish attacker-controlled content that immediately becomes Copilot's source of truth for that query.

Why This Matters: The Trust Escalation

When Copilot answers a question, users perceive the response as having Microsoft's implicit endorsement. Unlike a raw Google search where users evaluate multiple results, Copilot synthesizes a single answer from retrieved sources. Users who receive a Copilot response:

• Are less likely to independently verify the source.
• See a confident, well-formatted answer rather than a list of links to evaluate.
• May act on the information directly (e.g., following a "security policy" or executing a "recommended command").

This trust differential is what makes data void exploitation through Copilot qualitatively more dangerous than conventional search engine manipulation.

Key Findings

▶ Watch: Technique: key term association attack to inject content into Copilot answers (15:46)

1. Copilot consistently uses Bing's top-ranked result as its primary source for queries where training data is insufficient — verified empirically by Diehl across dozens of test queries.

1. Attacker-created content targeting a specific data void can achieve top Bing ranking without significant SEO investment — the absence of competing authoritative content is sufficient.

1. Copilot will synthesize and present attacker-controlled content as authoritative, including following attacker-supplied instructions embedded in the content, in certain prompt injection-susceptible configurations.

1. The data void C2 model is feasible: Diehl demonstrated that instructions embedded in attacker-controlled web content indexed by Bing can be retrieved and relayed by Copilot to a target, functioning as a one-way (or potentially bidirectional) C2 channel using Microsoft infrastructure as the communication medium.

1. No persistence on the target machine is required for the initial injection phase — the attack surface is the user's interaction with Copilot, not a traditional malware foothold.

1. Microsoft bug bounty and MV Researcher recognition: The vulnerability class was accepted by Microsoft's security response team, validating its severity.

Technical Deep Dive

▶ Watch: Demo breakthrough: Copilot executes attacker instructions from retrieved page (24:42)

Phase 1: Identifying and Targeting a Data Void

Diehl's methodology for finding usable data voids:

1. Identify a query that a target user is likely to ask Copilot — ideally one where they would act on the answer without verification. Examples: how to perform a specific corporate IT procedure, how to configure a specific internal tool, what a company's password reset policy says.

1. Verify the data void: Search Bing directly for the target query. If the top results are thin, unrelated, or absent — the void exists.

1. Generate attacker-controlled content: Use an AI writing tool (Diehl used AI to generate the content) to create a plausible, well-formatted webpage that appears to be an authoritative source on the query. The content should be semantically relevant to the query to achieve high Bing ranking.

1. Publish and index the page: Deploy the page to a publicly accessible domain. Submit it to Bing's URL submission tool or simply wait for natural crawling. For a genuine data void, ranking near the top can happen within days or hours.

Phase 2: Content Injection into Copilot Answers

Diehl demonstrated this concretely with a fabricated financial policy he called the "Guilt Gift Policy" (translating to "Geld Gift Richtlinie" in German, literally "money poison policy"). The page described a fictional money laundering procedure — obviously illegal content that no legitimate organization would document. He published the page, submitted it to Bing, and asked Copilot questions that would surface it.

Result: Copilot retrieved the attacker-created page and presented the money laundering procedure as an answer, citing the attacker's website as the source with apparent confidence.

This demonstrated Copilot's vulnerability to simple data void exploitation without any prompt injection or technical manipulation — just SEO-targeting a void.

Phase 3: Prompt Injection via Retrieved Content

Beyond simply surfacing false information, Diehl explored using the retrieved content as a prompt injection vector — embedding instructions for Copilot in the attacker-controlled page text, directing Copilot to:

• Modify its answer format or content.
• Suppress source citations.
• Include specific content in its response regardless of the user's actual query.
• Relay output back to an attacker-controlled endpoint via crafted hyperlinks in the response.

Prompt injection via retrieved web content exploits the fact that Copilot processes retrieved page text in a context where it can be mistaken for instructions. The mitigation for this specific sub-class is distinct from the data void problem: it requires Copilot's retrieval pipeline to sanitize or isolate retrieved content from the instruction context.

Phase 4: The C2 Channel

The most novel and alarming finding: using the data void and prompt injection combination as a C2 delivery mechanism for implants on a target machine. The attack model:

1. An implant on the target machine monitors Copilot interactions (either by hooking the Copilot client, reading clipboard, or intercepting the API calls from a browser extension context).
2. The attacker updates the content of the data void page to include encoded C2 instructions.
3. When the target user (or an automated system) queries Copilot with a trigger query, Copilot retrieves the attacker's updated page.
4. The injected content in Copilot's response is read by the implant as a C2 command.
5. The implant executes the command, and results can be exfiltrated through a similar web-based channel.

The result: C2 traffic that travels as legitimate HTTPS connections to microsoft.com (Copilot API) and bing.com (search retrieval), with no direct connection from the implant to attacker infrastructure. From a network monitoring perspective, the traffic is indistinguishable from an employee using Microsoft Copilot for research.

Bing's Role and Responsibility

Diehl acknowledged that the root issue is partly in Bing's ranking algorithm — the data void problem exists because Bing indexes and ranks attacker content in the absence of authoritative alternatives. Potential platform-level mitigations include:

• Requiring a minimum domain age, reputation score, or backlink threshold before newly published content ranks highly for novel queries.
• Flagging queries that match known data void characteristics and refusing to surface low-authority content in those cases.
• Implementing stricter content quality signals for content that will be used as Copilot grounding.

However, Diehl noted that these mitigations must balance against Bing's legitimate need to surface timely, newly-published content for legitimate queries about recent events.

Demo / Proof of Concept

▶ Watch: C2 channel: Copilot becomes an attacker-controlled command-and-control agent (29:18)

The demonstration proceeded in two stages:

Stage 1 (Disinformation demo):

• Diehl queried Copilot about the "Guilt Gift Policy" — a query that had no prior results.
• Copilot retrieved the attacker's fabricated page and synthesized an answer describing the money laundering procedure in formal policy language.
• The demo showed the Copilot response alongside the Bing search result to confirm the source.

Stage 2 (C2 channel demo):

• A simulated "implant" (a simple monitoring script) was shown watching Copilot's responses for a trigger phrase.
• The attacker updated the indexed page to include a base64-encoded command.
• A user queried Copilot, received the updated answer containing the encoded command.
• The monitoring script decoded and "executed" the command (demonstration: open a calculator), confirming the channel's operability.
• Network capture showed the implant's traffic consisted solely of standard HTTPS connections to Microsoft and Bing endpoints.

Defensive Implications

▶ Watch: Responsible disclosure: Microsoft's response and the patch timeline (38:18)

For Organizations Using Copilot

• User awareness training: Train users to treat Copilot answers about internal procedures, policies, or configurations with healthy skepticism, particularly for topics not well-represented in public documentation. Verify against authoritative internal sources.
• Evaluate Copilot for internal document grounding: Configure Copilot to prioritize internal documents and SharePoint content over public web retrieval for internal queries. The enterprise Copilot configurations that limit or disable Bing-backed retrieval in favor of organizational data reduce this attack surface significantly.
• Monitor for anomalous Copilot usage: If Copilot is deployed in automated or semi-automated workflows, audit what queries are being made and what responses are being acted upon.

For Microsoft

• Improve retrieval grounding quality: Implement domain authority and content age signals more aggressively in the retrieval pipeline that feeds Copilot.
• Isolate retrieved content from instruction context: Implement retrieval content sandboxing to prevent injected instructions in fetched web pages from influencing Copilot's behavior.
• Implement query-level data void detection: Identify queries targeting data voids and either refuse to answer from low-authority sources or add uncertainty qualifiers to the response.

For Red Teams

This research provides a new post-exploitation evasion technique: using Copilot as a C2 relay eliminates traditional C2 infrastructure detection patterns. Red teams should add this to their evasion toolkit for assessments of environments where Microsoft Copilot is deployed.

Key Takeaways

• Copilot's Bing-backed retrieval architecture can be exploited by publishing attacker-controlled content into a data void — a topic with no existing authoritative web content.
• Copilot presents attacker-controlled content with Microsoft-level implicit trust, making recipients less likely to verify or question it.
• Prompt injection via retrieved content can direct Copilot's behavior beyond simple misinformation delivery.
• The attack chain enables a C2 channel that blends into legitimate Copilot API traffic, bypassing traditional network-based C2 detection.
• No foothold is needed for the data void injection phase — the attack requires only that the target user interacts with Copilot about a topic the attacker has seeded.
• Microsoft recognized the research through its Most Valuable Researcher program, validating both the severity and the novelty of the findings.
• Enterprise Copilot configurations that prioritize internal document retrieval over public Bing search substantially reduce this attack surface.

About the Speaker

Tobias Diehl is an offensive security engineer with over five years of experience working in the financial industry. He also actively participates in bug bounty programs and was selected as one of Microsoft's Most Valuable Researchers in the year of this disclosure. His research focuses on AI system security, including prompt injection, retrieval-augmented generation (RAG) vulnerabilities, and the abuse of AI assistant trust models. He encourages connection through his published social media handles and is known as an accessible and collegial member of the security research community.