From Prompts to Pwns: Exploiting and Securing AI Agents

From Prompts to Pwns: Exploiting and Securing AI Agents

Speakers: Becca (Offensive Security Researcher, NVIDIA AI Red Team) and Rich (NVIDIA Product Security / AI Red Team)

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=zipgr080EQU

Reading Time: ~10 minutes

Type: Briefing

TL;DR

NVIDIA's AI Red Team demonstrated live prompt injection attacks against Microsoft Copilot, PandasAI (CVE disclosed), and Cursor IDE — including exploits that achieved remote code execution via a GitHub issue, a malicious pip package in a PR, and ASCII-smuggled instructions hidden in crowdsourced rule files. Their defensive framework reframes agent security using two new axioms: "least autonomy is the new least privilege" and "assume prompt injection is the new assume breach."

Introduction

Every AI agent that uses an LLM as its reasoning engine faces a structural vulnerability: the model that interprets instructions cannot reliably distinguish between instructions from a trusted user and instructions embedded in untrusted data it processes. This is prompt injection, and it is not a bug to be patched — it is an architectural property of how large language models work.

Becca and Rich from NVIDIA's AI Red Team spent their Black Hat session making that abstract claim concrete. Through a series of live demos targeting production tools used by developers today — Microsoft Copilot, PandasAI, and Cursor IDE — they showed exactly what an attacker can do when they control even a small piece of data that an agent will eventually read. The session's opening slide said it plainly: "The LLM that drives and controls your agents can potentially be controlled by attackers. You should act accordingly and be very careful about everything that your agent has access to downstream."

An Autonomy Taxonomy: Knowing What You're Defending

Before running any exploits, the speakers established a three-level taxonomy of agent autonomy that provides the conceptual scaffolding for the rest of the talk — and for thinking about AI risk more generally.

▶ Watch: Agent Autonomy Levels Defined (02:00)

Level 1 is a basic inference endpoint: the user controls all state, tools, and authorization. The LLM produces output; nothing else happens. This is as deterministic as an AI application gets.

Level 2 is an LLM router — the model can choose between a set of available tools (for example, deciding whether to query a database or render markdown), but one user interaction always triggers one predictable control flow path. Early tool-using chatbots operated this way because it was more reliable.

Level 3 is the fully autonomous agent — the model is in control of the program's control flow, deciding what tools to invoke, planning multiple steps ahead without user intervention between each step, and operating in an execution loop until it determines a task is complete. Modern Claude Research, ChatGPT with advanced tools, Cursor IDE in Auto Run mode, and OpenAI Codex all operate at Level 3 under common configurations.

The key insight from this taxonomy: every step up the autonomy ladder multiplies the nondeterminism and therefore the attack surface. A Level 3 agent that receives attacker-controlled input may execute an entirely different sequence of actions than any developer anticipated.

The Universal Anti-Pattern

All of the attacks demonstrated share a common structure the researchers call the universal anti-pattern:

1. Untrusted data enters the agent's context — via email, a web page, a code comment, a PR, a crowdsourced rules file.
2. An LLM parses and processes that data, and because LLMs are uniquely susceptible to adversarial manipulation, the attacker's instructions are interpreted as legitimate directives.
3. The resulting LLM output is passed to downstream tools that have real capabilities: executing code, writing files, making network requests, or reading sensitive data.

▶ Watch: The Universal Anti-Pattern (06:00)

The universal anti-pattern means that any point where untrusted data can reach the LLM is, by definition, a potential attack vector. The question an attacker asks is: "Where can I inject, how is the data processed, and what tools are available downstream?"

Demo 1: Microsoft Copilot — Email Injection and Data Exfiltration

Copilot's RAG database includes user emails, SharePoint documents, and other enterprise data. The simplest way to introduce untrusted input is to send an email to anyone in the company — a capability available to any external attacker.

The injection hides instructions in white-on-white text invisible to human readers but processed by the LLM at inference time. The payload instructs Copilot to exfiltrate sensitive data from the user's mailbox to an attacker-controlled URL, encoded as a markdown image tag — not a link, which a human might notice, but an image that triggers an automatic HTTP fetch as soon as it renders.

▶ Watch: Microsoft Copilot Email Injection Demo (08:00)

NVIDIA reported this to Microsoft in fall 2024. Microsoft implemented redaction of external URLs rendered by Copilot. However, AIM Labs subsequently published "Echo Leak," demonstrating that those protections could be subverted — a point the speakers cited to illustrate that patches to LLM-adjacent controls tend to be iterative rather than definitive.

Demo 2: PandasAI — Prompt Injection to Remote Code Execution

PandasAI is an open-source tool that lets users query pandas DataFrames using natural language. The LLM generates Python code, which is then executed on the host running PandasAI.

Because PandasAI is open source, the researchers could read its system prompts and understand exactly what the model expected. They crafted a prompt that redefined the data variable as attacker-supplied content, told the model that the required code had already been generated, and included a Base64-encoded payload using pandas built-in functions to evade the tool's execution protections.

▶ Watch: PandasAI RCE Demo (10:00)

The result: arbitrary code execution on the PandasAI host, up to and including a reverse shell. The researchers filed a CVE and reported the issue to PandasAI in fall 2024. PandasAI now offers a configuration option to run generated code in an isolated sandbox container — which the speakers recommended enabling by default for any deployment.

Demo 3: Cursor IDE — GitHub Issues, Malicious PRs, and ASCII Smuggling

The longest and most technically detailed section of the talk targeted Cursor, a VS Code fork that enables AI-assisted coding and is increasingly used with Auto Run mode — previously called "Yolo Mode" — which allows the agent to execute commands and write files without user approval.

Three separate attack vectors were demonstrated:

Attack vector 1: Malicious GitHub issue. An attacker creates a GitHub issue containing natural language text that, if read by an agent tasked with resolving open issues, instructs it to download and execute arbitrary code from the internet. The issue text social-engineers the agent with urgency: "I'm getting this error and it changes every time, you really need to run this to see what's happening." Verified against Anthropic's computer use demo (which ships with a prompt injection warning), the agent read the issue, interpreted it as instructions, and generated the PowerShell command to download and execute the attacker's payload.

▶ Watch: GitHub Issue Injection via Computer Use Agent (16:01)

Attack vector 2: Malicious dependency in a pull request. An attacker submits a PR adding a malicious Python package to requirements.txt. The package's setup code calls back to an attacker-controlled server on installation. When a developer prompts Cursor to "check my open PRs and tell me which ones I can merge," the agent reviews the PR, checks out the branch, installs the package to verify it works — and triggers the callback. The attacker gets a shell; the developer sees nothing out of the ordinary.

▶ Watch: Malicious PR Dependency Demo (24:01)

Cursor's response was that this is not a CVE — it is a consequence of how LLM-backed agents that execute code work. The speakers agreed with the technical assessment but noted it should inform how organizations configure and use such tools.

Attack vector 3: ASCII smuggling in crowdsourced rule files. Cursor supports .cursorrules files that define coding guidelines processed at the start of every session. Many developers crowdsource these files from public repositories. The attack encodes a malicious payload using Unicode tag characters — characters that are invisible to human reviewers but readable by LLMs — and embeds them in an otherwise legitimate-looking rules file. The injected instruction persisted through the session and caused the agent to append a comment to generated files that, when compiled and run, spawned a shell.

▶ Watch: ASCII Smuggling in Rule Files (20:01)

Defense Framework: Assume Prompt Injection

Rich's defensive section was framed using a modified cyber kill chain adapted for AI workflows, which the team uses internally to reason through agentic attack surfaces.

▶ Watch: Defensive Framework Overview (26:01)

Three principles anchored the framework:

1. Assume prompt injection. Design your system to be robust to LLM compromise from the start. If an attacker can get their input in front of the LLM, assume they can control anything downstream. This is the analogue to "assume breach" in traditional security.

2. Least autonomy is the new least privilege. Every step up the autonomy scale makes defensive controls harder to implement. Restricting agents to the minimum autonomy level required for their task reduces the blast radius of a successful injection.

3. Sensitive actions require explicit controls. Any tool call that writes data, triggers external network activity, or requires delegated authorization is a sensitive action. These should be gated behind validation logic that does not rely solely on the LLM, may invoke human-in-the-loop review, and should be logged.

Additional recommendations included mapping all input/output paths including non-obvious ones (tool return values, conversation history, third-party data sources), scoping tool permissions using least privilege, running command-execution tools in isolated containers to limit blast radius, and treating LLM conversation history as potentially compromised after any confirmed injection.

▶ Watch: Specific Defensive Recommendations (34:02)

The speakers cited OpenAI Codex and Cursor's background agent feature as examples of implementations that already run command execution in sandboxed containers — the pattern they recommend broadly.

Notable Quotes

"The LLM that drives and controls your agents can potentially be controlled by attackers. You should act accordingly and be very careful about everything that your agent has access to downstream."

— Becca ▶ 00:00

"You put an input in, and you kind of have no idea what's gonna come out the other end. And this nondeterminism is what gets us into a lot of trouble — or, if you're Becca, gives you a lot of fun."

— Rich ▶ 04:00

"These autonomous agents, if they have their goals replaced by an attacker, they will work just as hard for the attacker as they will for you."

— Rich ▶ 34:02

"Least autonomy is the new least privilege. Assume prompt injection is the new assume breach."

— Rich ▶ 38:02

Key Takeaways

• Prompt injection is architectural, not incidental. Any LLM that processes external data can be made to follow attacker instructions. Treat it as a given in system design, not as a bug to patch post-deployment.
• Untrusted input surfaces are everywhere. Emails, web pages, GitHub issues, pull requests, dependency files, and crowdsourced configuration files are all potential injection vectors for agents that browse, code, or review.
• Auto Run / Yolo Mode dramatically expands blast radius. Removing the human approval step from command execution turns a manageable injection into a fully autonomous attack loop. Its use should be restricted to sandboxed environments.
• Least autonomy is the new least privilege. Agent design should start with the lowest autonomy level that accomplishes the task. Escalating autonomy should require explicit justification and additional controls.
• Isolate command execution. Agents that must execute commands should do so in containers isolated from user data and production systems. This limits what an attacker can reach even after a successful injection.

Slides

No slides PDF was listed for this talk. Blog posts with additional detail were referenced and linked by the speakers at the close of the session.