Poisoning Fast and Slow

Poisoning Fast and Slow

Speakers: Emerson T. Brooking, Director of the Digital Forensic Research Lab, Atlantic Council

Conference: CYBERWARCON

YouTube: https://www.youtube.com/watch?v=t_lnV3Hqcwk

Overview

In an era increasingly shaped by artificial intelligence, the integrity of information disseminated by large language models (LLMs) has become a critical concern. Emerson T. Brooking, Director of the Digital Forensic Research Lab (DFR Lab) at the Atlantic Council, delivered a compelling talk at CYBERWARCON, dissecting the intricate and evolving intersection of influence operations (IO) and LLM poisoning. This presentation, a culmination of efforts from the DFR Lab's 20 analysts, served as a stark warning about the sophisticated and often subtle methods threat actors are employing to manipulate the information landscape through AI.

Brooking introduced a crucial distinction: "fast poisoning" and "slow poisoning." Fast poisoning refers to immediate, often opportunistic attempts to compromise the external retrieval mechanisms of LLMs, influencing real-time responses. Slow poisoning, on the other hand, aligns with more traditional data poisoning techniques, targeting the core training data of AI models for long-term, systemic manipulation. The talk underscored why this distinction matters, particularly as less sophisticated influence actors increasingly leverage the former, while state-backed entities continue to pursue the latter, with profound implications for national security and public discourse.

The core message resonates deeply within the cybersecurity community: LLMs, despite their advanced capabilities, are not immune to manipulation. Their reliance on vast datasets and external information retrieval makes them potent, yet vulnerable, vectors for influence operations. Brooking’s analysis highlighted that the danger isn't merely theoretical; evidence suggests active campaigns are already underway, subtly reshaping the "world model" of these powerful AI systems and, by extension, the information consumed by millions.

Background

▶ Watch: Introduction to LLM poisoning and influence operations (0:00)

Data poisoning, in the context of AI and LLMs, is defined by IBM as "a type of cyber attack where threat actors manipulate or corrupt the training data used to develop AI models." Historically, this concept often invoked images of direct compromise of core training datasets, a complex endeavor typically requiring high levels of sophistication. A common example of a backdoor attack in practice involves injecting specific triggers into training data that, when encountered in a query, can cause a model to output a manipulated or broken response, such as an arbitrary XML tag.

A significant development in understanding the potency of these attacks came from a recent report by Anthropic and the UKAI Institute. Their findings revealed that poisoning even a large language model with billions of parameters does not require an overwhelming amount of corrupted data. Specifically, they found that it requires a "near constant number of documents"—approximately 250 corrupted documents—for a model with 13 to 17 billion parameters, representing a minuscule 0.00016% of total training tokens, to significantly alter its behavior. This discovery dramatically lowers the barrier to entry for effective data poisoning.

However, Brooking noted that many influence operation (IO) actors lack the sophistication or direct access required to penetrate the servers of major AI model companies to corrupt core training data. Consequently, these actors have pivoted their strategies. Instead of compromising the foundation, they are increasingly targeting the external retrieval mechanisms of LLMs. When users enable web query functions in models like ChatGPT, the LLM retrieves information from external sources. These sources—news articles, social media, forums, and even Wikipedia—become the new battleground for IO.

This phenomenon has led to a debate over terminology. While some have dubbed it "LLM grooming," Brooking argues that it is still fundamentally a form of data poisoning. He proposes the terms "fast poisoning" for immediate, retrieval-based attacks and "slow poisoning" for the more traditional, long-term corruption of core training data. Several organizations, including the DFR Lab, Check First (based in Finland), the American Sunlight Project, and NewsGuard, have independently published analyses this year confirming the existence of a massive pro-Kremlin propaganda network. This network is reportedly churning out "tens of thousands" of fake news articles each month, which are then working their way into Wikipedia citations and, subsequently, into the information LLMs retrieve and cite. This real-world example provides critical context for the immediate threats posed by fast poisoning.

Key Findings

▶ Watch: Grok's contradictory answers on a viral image (4:40)

Brooking's presentation unveiled several critical findings regarding the susceptibility of LLMs to influence operations, categorized by the "fast and slow" poisoning paradigm:

1. Low Barrier to Entry for Effective Poisoning: Building on the Anthropic/UKAI Institute research, Brooking reiterated that even a minute amount of corrupted data (as low as 0.00016% of total training tokens, or approximately 250 documents for a 13-17 billion parameter model) can fundamentally alter an LLM's responses. This demonstrates a significant vulnerability that lowers the technical expertise required for impactful poisoning.

1. External Retrieval Mechanisms are a Primary Attack Surface for "Fast Poisoning": For less sophisticated IO actors, direct compromise of LLM training data is challenging. Instead, they actively target the public web sources that LLMs use for real-time information retrieval. This "fast poisoning" method allows immediate manipulation of model outputs, as observed in campaigns like the pro-Kremlin network feeding fake news into Wikipedia and subsequently into LLM citations.

1. Community Moderation Systems are Emerging Attack Surfaces: Platforms like X's (formerly Twitter) Community Notes, designed for crowdsourced fact-checking, can be exploited. Brooking highlighted evidence, including analyses from Active Fence, showing "near-identical text recycling" and "placeholder tokens" suggesting coordinated attempts to brigade Community Notes. Successfully brigading these notes can rapidly alter an LLM's "world model" for fast-moving events, as demonstrated with the Grok case study.

1. AI Checking AI is Flawed and Vulnerable to Manipulation: The reliance on AI-powered moderation (e.g., AI-generated Community Notes) to combat misinformation introduces new risks. Brooking presented an example where Grok's AI-driven analysis incorrectly insisted a clearly fabricated video of goats charging down a mountain was true, particularly in non-English contexts. This illustrates the inherent limitations and potential for AI to be misled, even when attempting to self-correct.

1. "Slow Poisoning" Narratives are Infiltrating Core Training Data and Will Manifest Over Time: Campaigns like "Operation Pravda" (pro-Kremlin) and "PRC Glass Bridge" (Chinese state-backed) are systematically injecting propaganda into public datasets like Common Crawl, a foundational source for many LLM training datasets. Due to the lag in LLM retraining cycles, these hostile narratives are expected to increasingly manifest in open-source models as more time passes, posing a long-term, systemic threat.

1. Non-English Language Contexts are Particularly Vulnerable: LLMs are generally weaker in non-English languages. This diminished capability amplifies the threat of poisoning, as manipulated content in these languages may be less effectively identified and countered by the models themselves or human moderators.

1. The Rise of "Generative Engine Optimization" (GEO): Brooking concluded by predicting the emergence of a new industry focused on optimizing content for generative AI, akin to traditional Search Engine Optimization (SEO). This new frontier, "Generative Engine Optimization," will inevitably attract threat actors seeking to monetize or weaponize influence, creating a new and evolving landscape of cyber threats.

Technical Deep Dive

▶ Watch: Community Notes as a new attack surface (6:00)

The technical underpinnings of LLM poisoning, as detailed by Brooking, span both immediate, external manipulations and long-term, foundational compromises. Central to understanding the core vulnerability is the finding by Anthropic and the UKAI Institute: a remarkably small amount of poisoned data can have a disproportionately large effect. For models with 13 to 17 billion parameters, as few as 250 corrupted documents, constituting a mere 0.00016% of total training tokens, are sufficient to fundamentally alter the model's behavior. This means that a relatively minor, targeted injection of malicious data can reorient an LLM's understanding or response to specific queries, making the attack surface much broader than previously assumed for core training data.

Brooking elaborated on fast poisoning using Grok on X (formerly Twitter) as a prime example. Grok is significant because it's a "public-facing foundation model" where most user queries and conversations are open and discoverable. The DFR Lab is actively scraping all Grok responses—an estimated 200 posts per minute, or 300,000 posts daily—to analyze its behavior. Users, Brooking noted, often incorrectly perceive Grok as omniscient, making its responses highly influential.

A case study from June illustrated this vulnerability: a viral image purported to show an Iranian missile strike on Tel Aviv airport. As users turned to Grok for verification, the model initially provided contradictory answers within the same second. However, after the first Community Note related to the image was posted, Grok's answers suddenly consolidated around the information in that note. This phenomenon spotlights Community Notes as a critical attack surface for fast poisoning. While Community Notes are designed as a "surprisingly durable, crowdsourced moderation system" that requires consensus from users with unalike viewpoints, they are not impervious. Active Fence analysis, cited by Brooking, has observed "increasing frequency of near-identical text recycling and evidence of placeholder tokens," suggesting organized attempts to brigade these notes. If successful, brigading Community Notes for a fast-moving event can effectively "change the world model of Grok" as it answers subsequent user queries globally.

Further complicating the defensive landscape is the emergence of AI checking AI. X has begun deploying AI-powered Community Notes, ostensibly to moderate content more rapidly. However, Brooking presented a clear failure case: an "AI community notes mechanism" on X incorrectly insisted that a "frenetic, but pretty clearly not real, video of some goats charging down a mountain" was true, particularly in its Spanish-language analysis. This demonstrates that even AI-driven moderation can be fallible and potentially susceptible to sophisticated manipulation, or simply prone to error, especially in non-English contexts where LLMs are generally weaker.

Turning to slow poisoning, Brooking described the challenge of analyzing proprietary LLMs. "If you can't look in an LLM's stomach," he quipped, "you can still visit its favorite restaurant." This analogy refers to analyzing Common Crawl, a massive, publicly available web corpus that serves as a "source of truth for a lot of LLM development." The DFR Lab's ongoing project involves scrutinizing Common Crawl for evidence of propaganda from campaigns like "Operation Pravda" (pro-Kremlin) and "PRC Glass Bridge" (Chinese state-backed). They found that articles from entities like RT and Chinese news wires were manifested "multiple times" within Common Crawl's dataset, often under various domains due to their nature as news wires.

These repeated inclusions in Common Crawl mean that hostile narratives are effectively being queued for ingestion into LLM training datasets. Brooking highlighted that because there's a "lag with open source models in their training data," the "manifestation of some of these hostile and foreign state propaganda narratives" will only increase as time passes. This suggests a ticking time bomb where past influence operations, already embedded in foundational datasets, will slowly but surely corrupt the long-term knowledge base of future and updated LLMs. The implications extend beyond state actors, as this method of manipulating generative AI for specific outcomes is rapidly becoming a new industry: Generative Engine Optimization (GEO), a lucrative frontier for both legitimate businesses and malicious actors.

Demo / Proof of Concept

▶ Watch: 'Any amount of poison causes harm' principle (9:40)

The talk did not feature a live demonstration or a specific proof-of-concept tool. Instead, Emerson Brooking presented analytical findings, case studies, and observational evidence derived from the ongoing research of the DFR Lab and other organizations. The evidence included specific examples of Grok's behavior, references to reports by Anthropic, the UKAI Institute, and Active Fence, and insights from the DFR Lab's projects analyzing Grok responses and Common Crawl data.

Defensive Implications

▶ Watch: Generative Engine Optimization: next frontier of influence (10:10)

The insights shared by Emerson Brooking offer crucial guidance for defenders across various domains, from AI developers to everyday users. Mitigating the dual threats of fast and slow poisoning requires a multi-faceted approach:

For LLM Developers and Providers:

• Strengthen Ingress Filtering for Retrieval Mechanisms: Implement more robust filtering and validation processes for external data sources accessed by LLMs, especially those used for real-time web queries. This includes advanced anomaly detection, source reputation scoring, and content verification to identify and block propagandistic or manipulated information before it can influence responses.
• Enhance Transparency in Training Data: While proprietary models may guard their full training datasets, more transparency regarding the major sources and versions of data used for training would enable external scrutiny and help identify potential poisoning vectors. For open-weight models, detailed documentation of training data provenance is essential.
• Invest in Continuous Anomaly Detection for Training Data: Develop and deploy sophisticated AI-driven systems to monitor training datasets for statistical anomalies, patterns indicative of coordinated content injection, and known propaganda signatures, even in minute quantities (e.g., the 0.00016% threshold identified by Anthropic).
• Fortify Community Moderation Systems: For platforms like X that rely on crowdsourced moderation, invest in better algorithms and human oversight to detect and counter brigading attempts, "text recycling," and the use of "placeholder tokens." The integrity of these systems directly impacts LLMs that may reference them.
• Prioritize Non-English Language Defenses: Recognizing that LLMs are "considerably weaker" in non-English contexts, developers must allocate more resources to improve language capabilities, enhance content moderation, and implement poisoning detection specifically for these languages, which are often targets of state-sponsored influence operations.
• Develop AI-Resilient AI Moderation: The failure of AI to correctly identify the "goats charging down a mountain" video highlights the need for more robust and context-aware AI moderation tools that are less susceptible to subtle manipulation or inherent biases, particularly when AI is checking AI.

For End-Users and the Public:

• Cultivate Critical Information Literacy: Users must be educated to approach LLM outputs with skepticism, particularly concerning fast-moving or politically charged events. LLMs are not "omniscient" and can reflect biases or manipulations present in their training or retrieval data.
• Verify Information Across Multiple, Reputable Sources: Encourage the practice of cross-referencing LLM-generated information with established, diverse, and credible news organizations, academic sources, and fact-checking websites.
• Understand LLM Limitations: Users should be aware that LLMs can hallucinate, be influenced by propaganda, and may struggle with nuance, especially in less-resourced languages or on rapidly evolving topics.

For Researchers and the Security Community:

• Monitor Foundational Datasets: Continue active monitoring and analysis of public datasets like Common Crawl for the infiltration of hostile narratives and propaganda, as these represent future "slow poisoning" vectors for LLMs.
• Research Generative Engine Optimization (GEO): Proactively research the emerging field of GEO to understand its legitimate applications and anticipate how threat actors will exploit it for malicious purposes, developing countermeasures ahead of widespread abuse.
• Focus on Open-Weight Model Analysis: Leverage the greater transparency of open-weight LLMs to conduct in-depth forensic analysis of their training data and output behaviors, which can provide insights into general LLM vulnerabilities.
• Foster Cross-Organizational Collaboration: Continue and expand collaboration between organizations like the DFR Lab, Check First, American Sunlight Project, and NewsGuard to identify, track, and publicly expose influence operations targeting the LLM ecosystem.

Key Takeaways

• LLMs are a New Frontier for Influence Operations: Generative AI models are highly susceptible to both immediate and long-term manipulation by state and non-state actors, posing significant risks to information integrity.
• "Fast" and "Slow" Poisoning Require Distinct Defenses: Fast poisoning targets real-time retrieval mechanisms (e.g., web queries, community notes), while slow poisoning corrupts foundational training data, with both demanding specific mitigation strategies.
• Minimal Data Corruption Can Have Disproportionate Impact: As little as 0.00016% of corrupted training tokens (or ~250 documents for a 13-17B parameter model) can fundamentally alter an LLM's behavior, lowering the barrier for effective attacks.
• Community Moderation Systems are Emerging Attack Surfaces: Crowdsourced platforms like X's Community Notes can be brigaded to influence LLM responses, highlighting the need for robust defense of these critical information gatekeepers.
• AI Checking AI is Not a Panacea: Reliance on AI-powered moderation can introduce new vulnerabilities and errors, especially in non-English contexts where LLMs are weaker, necessitating human oversight and more resilient AI design.
• "Generative Engine Optimization" (GEO) is an Emerging Threat and Industry: The drive to optimize content for generative AI creates a new landscape for both legitimate business and malicious influence, demanding proactive security measures.

About the Speaker(s)

Emerson T. Brooking is the Director of the Digital Forensic Research Lab (DFR Lab) at the Atlantic Council. In this role, he leads a team of 20 analysts dedicated to understanding and countering emerging threats in the digital information environment, including the complex interplay of influence operations and artificial intelligence. Prior to his work at the Atlantic Council, Brooking served as a cyber policy advisor at the office of cyber policy at the Department of Defense (DOD), bringing a strong background in national security and cyber warfare to his analysis of information manipulation.