Threat Prompt: AI Security

Threat Prompt: AI Security

Speakers: Craig Balding, Independent Consultant

Conference: Hacktivity

YouTube: https://www.youtube.com/watch?v=IFRX6rUDXyY

Overview

In his Hacktivity talk, "Threat Prompt: AI Security," independent consultant Craig Balding delivers a compelling exploration of the rapidly evolving landscape of artificial intelligence from a cybersecurity perspective. The presentation serves as both a primer for security professionals needing to get up to speed on AI fundamentals and a practical guide to leveraging AI for defensive tasks, all while highlighting the emerging security challenges. Balding emphasizes that understanding AI is no longer optional for those in cybersecurity, positioning it as a fundamental shift akin to previous technological revolutions.

Balding's talk is particularly relevant given the explosive growth and adoption of generative AI tools like ChatGPT. He delves into the significant productivity gains AI offers across various professional domains, from document creation to software development, thereby underscoring why business leaders are so eager to integrate these technologies. However, this rapid adoption introduces novel attack surfaces and complex risks that traditional security models may not adequately address. The presentation meticulously dissects these risks, offering both conceptual frameworks and concrete examples of how AI can be exploited and, conversely, how it can be harnessed as a powerful ally for defenders.

The talk distinguishes between the broad field of artificial intelligence and its scientific underpinning, machine learning, acknowledging the long history of the latter while focusing on the current generative AI paradigm. Balding's pragmatic approach, rooted in hands-on experimentation, cuts through much of the prevailing hype, providing a realistic assessment of AI's capabilities and limitations. By outlining specific use cases and defensive implications, he equips the audience with actionable insights to navigate the security implications of AI in their organizations, stressing the critical need for security expertise in this burgeoning domain.

Background

▶ Watch: Introduction to AI security and talk agenda (0:00)

The current surge of interest in AI, particularly generative models, represents a significant inflection point in technology, driven primarily by unprecedented performance gains. Craig Balding highlights this by referencing key studies: MIT researchers found that professionals using ChatGPT completed writing tasks (memos, strategy documents) 37% faster with an increase in average writing quality. Similarly, a GitHub study indicated that developers using AI as a pair programmer completed tasks 55% faster. Such dramatic efficiency improvements make AI an irresistible proposition for businesses seeking to optimize operations and resource allocation.

Balding clarifies the distinction between Artificial Intelligence (AI) and Machine Learning (ML), positing AI as the broader concept encompassing the simulation of human intelligence, while ML is the scientific discipline that underpins many AI advancements. He notes that machine learning security has been a field for many years, particularly concerning data lakes and algorithms, where 80% of the job often involves data sourcing, collection, and cleaning. The current focus, however, is on the new capabilities brought by generative AI in text, code, and images, exemplified by the rapid user acquisition of ChatGPT.

Despite the hype, Balding acknowledges the "Gartner hype curve" phenomenon, noting a nascent cynicism around AI. He also touches upon the long history of AI, mentioning ELIZA, a natural language processing computer program developed between 1964 and 1966, as an early example of conversational AI. This historical context grounds the current advancements, demonstrating that the pursuit of artificial intelligence is not new, but its recent breakthroughs in generative capabilities are.

However, these new capabilities introduce significant challenges. Balding points out issues with reproducibility in AI services, where the same prompt can yield different results over time due to continuous model updates. This lack of consistency poses a problem for security decision-making, where predictable outcomes are paramount. Furthermore, the "black box" nature of many AI models clashes with regulatory demands for transparency and explainability, such as those from the EU, which require justifications for decisions made by AI (e.g., credit denials). Balding’s personal journey into understanding AI stems from a belief that comprehending other technical domains makes one a "better attacker and a better defender," a philosophy he extends to the imperative for security professionals to grasp AI.

Key Findings

▶ Watch: Challenges: AI reproducibility, control, and explainability (11:00)

Craig Balding's talk underscores several critical findings regarding the intersection of AI and security, highlighting both the opportunities and the inherent risks that define this new technological frontier.

Firstly, AI stands as a transformative force for productivity. The evidence from MIT and GitHub studies, showing 37% faster document creation and 55% faster coding with AI assistance, demonstrates an undeniable shift in operational efficiency. This economic imperative ensures rapid AI adoption across industries, making its security implications a top-tier concern.

Secondly, the talk identifies the emergence of novel and significant attack surfaces. Balding explicitly likens prompt injection to the "SQL injection of AI," emphasizing its ease of execution and the difficulty in securing against it. Another critical finding is model stealing, where attackers can train a smaller, cheaper model by querying a proprietary, expensively trained AI, effectively reverse-engineering its intelligence. These attacks represent fundamental vulnerabilities in the AI paradigm.

Thirdly, Balding highlights the increasing complexity of AI systems, moving beyond just the "model weights and biases" to encompass intricate data pipelines and broader deployment contexts. He uses the analogy of a "brain without the skull, connective tissue, or nerve endings" to describe a standalone model, stressing that real-world AI security involves securing the entire system, from data ingestion to output execution. The DarkBERT example illustrates the challenges of securing and cleaning specialized training data.

Fourthly, there is a clear trend towards democratized and localized AI. The discussion on running models on modest hardware like a Raspberry Pi, or more powerfully within a browser using WebGPU, indicates a future where AI processing moves closer to the user. This shift has implications for privacy, control, and the potential for personalized, domain-specific AI models trained on local data.

Fifthly, Balding anticipates the inevitable rise of AI monitoring AI. He posits that business leaders will naturally gravitate towards having one AI system oversee the outputs and behaviors of another, gradually reducing the need for human supervision. While efficient, this introduces a complex chain of trust and potential for cascading failures or misalignments that demand rigorous security attention.

Finally, a crucial finding is the urgent need for cybersecurity professionals in AI red teaming. Balding notes that current AI red teams often comprise individuals with strong data science backgrounds but lack hacking expertise. This gap presents a significant opportunity for security practitioners to apply their offensive security knowledge to probe and secure AI models and systems, an area he considers underserved.

Technical Deep Dive

▶ Watch: Understanding AI system architecture and attack surface (18:00)

Securing AI systems extends far beyond the core model weights and biases, which represent the "intelligence" derived from training. Balding emphasizes that a real-world AI deployment is a complex AI system architecture, likened to a "brain without the skull," requiring connective tissue and nerve endings to function within a broader context. This holistic view necessitates securing the entire pipeline, from data ingestion to model deployment and interaction.

A critical component of this architecture is the data pipeline. Balding highlights that "bad data in is bad data out," making data filtering and text pre-processing crucial, yet vulnerable, attack points. He references DarkBERT, an AI system trained on dark web data, as an example where extensive data cleanup was necessary, despite the inherent "badness" of the source data being integral to the model's intended purpose. The talk warns of significant risks associated with breaches of these massive training datasets, which can expose intellectual property or sensitive information, underscoring that some of the biggest AI risks lie outside the model itself, in the data handling.

The diversity of AI models is also discussed, categorizing them into encoder models, encoder-decoder models, and decoder models. Platforms like Hugging Face host a vast array of these models, allowing users to download "weights and biases" files to run locally, assuming sufficient hardware. This leads to a distinction between local AI and remote/hosted AI. Local execution, even on a Raspberry Pi (though slow, generating "one token every 10 seconds"), offers greater control and privacy. The emergence of WebGPU API in browsers like Chrome further pushes AI processing to the edge, enabling "just-in-time AI" inference without needing the model on the local machine or relying on external services. For professionals, Balding suggests a future with capable local AIs, fine-tuned on personal or organizational data, though he cautions against "naive chunking" of documents for training, advocating for domain-specific biasing to improve relevance and reduce "junk" output.

A key technical concept for AI interaction with the world is the UDA Loop (Observe, Orient, Decide, Act). This framework describes how AI moves from simply generating text to performing actions. In an "AI in a loop" scenario, the AI generates commands, executes them, observes the outcome (e.g., error messages), reorients itself based on feedback, decides on a new action, and acts again. This iterative process, which might involve giving the AI web access for research, necessitates robust sandboxing technologies. Sandboxes become critical to control "runaway AI" – whether due to malicious intent or simply generating "a lot of mistakes and doing bad things as a result."

Prompt engineering is presented as an iterative art, crucial for "steering" the AI to produce desired outcomes. The quality of the prompt (the instruction given to the AI) significantly impacts the result. Balding mentions "prompt stuffing" as a technique to overwhelm the AI with detailed instructions, biasing its output. The challenge of reproducibility, where the same prompt can yield different results from a constantly updated hosted AI, complicates consistent security analysis.

Regarding AI alignment, Balding points to InstructGPT as a standard approach used by hosted AI companies. This method involves a mix of human and machine supervision to align AI behavior with desired outcomes, with a gradual trend towards removing human oversight in favor of "one AI monitoring another AI."

Finally, the talk delves into specific attack vectors:

• Prompt Injection: Described as the "SQL injection of AI," it involves crafting malicious prompts that override or manipulate the AI's intended instructions, a vulnerability that is "not going to go away in a hurry."
• Model Stealing: This involves training a less expensive "student" model by querying a proprietary, highly trained "teacher" model. Balding notes that companies like OpenAI implement rate limiting and analyze query patterns to detect and prevent such intellectual property theft.
• Bias in Security Systems: Acknowledged as a "big can of worms," where inherent biases in training data can lead to unfair or inaccurate security decisions, mirroring societal biases.

These technical aspects underscore the vast and intricate attack surface presented by AI, demanding a multi-faceted and informed security approach.

Demo / Proof of Concept

▶ Watch: Use case: Transforming unstructured WHOIS data to JSON (34:00)

While Craig Balding describes these as "sample use cases" rather than formal demos, they effectively illustrate how AI can be practically applied in a security context, both for efficiency and for addressing specific challenges. His emphasis is on inspiring attendees to consider how AI can assist in their own work, cutting through the hype with realistic, tested examples.

1. Explaining Security Controls with Local AI (Concept):

Balding proposes a scenario where a local, capable AI could explain security controls to users. For instance, if a user selects a weak password, the AI could instantly rate it (e.g., "3 out of 10") and provide a detailed explanation of why it's weak and how to improve it. This addresses the problem of opaque security controls and user education. While acknowledging the risk of feeding sensitive data like passwords to a hosted AI (which he advises against), he highlights the potential of future local AIs for this purpose, also noting its utility for generating documentation for regulators or non-technical stakeholders.

1. Data Mining with GPT Plugins (Cloudflare Radar & Wolfram Alpha):

This use case demonstrates AI's ability to interface with external tools and perform complex data queries with simple natural language prompts. Using GPT plugins for Cloudflare Radar and Wolfram Alpha, Balding gave the prompt: "Plot the distribution of UDP in layer 34 attacks in Sweden over the past seven days." The AI successfully generated a chart, showcasing its capability to automate data mining tasks that would typically require writing specialized scripts or manual data aggregation. This highlights AI's potential as a "one-sentence" interface for cyber intelligence tasks.

1. Unstructured WHOIS Data to Structured JSON:

A common pain point in pen testing and red teaming is the inconsistent formatting of WHOIS data across different registrars. Balding showed how GPT-4, combined with Simon Willis's command-line tool llm, can convert unstructured WHOIS output into a perfectly formatted JSON record. The prompt explicitly defined the desired JSON structure. This exemplifies AI's power as "glue" for data transformation and automation, particularly for non-sensitive data, streamlining workflows and feeding structured data into other systems.

1. JavaScript AI Glossary Generator:

To demonstrate AI-assisted code generation, Balding tasked an AI with creating a JavaScript-based AI glossary for his website. Within about an hour, he had a reasonably well-functioning script. He notes that while he could have written it himself, the AI significantly reduced development time and shifted the debugging effort from his own errors to those introduced by the AI. This illustrates AI's utility in accelerating development for common, non-complex coding tasks.

1. Burp Suite BCheck Script Generation:

This is a particularly compelling technical example. Balding leveraged GPT-4 to generate BCheck scripts for Burp Suite, a web security scanner. The key insight was that the entire BCheck language specification fits within GPT-4's context window. By providing the specification along with a prompt like "generate a script that systematically figures out what cookies I can get rid of to send a minimum set," GPT-4 consistently produced accurate BCheck code. He noted that GPT-3.5 struggled, often generating Python instead of BCheck, underscoring the importance of model capability. This use case highlights AI's ability to generate code for novel or niche languages when provided with sufficient context, and emphasizes the value of "debating" the AI (e.g., asking for pros/cons or counter-arguments) to refine its outputs, even for code or architectural designs.

1. Privacy Policy Evaluator (Concept):

Balding described a system he built where a user pastes in a privacy policy, answers some questions, and the AI evaluates the policy against a set of predefined prompts, generating a full report in about 30 seconds. This demonstrates AI's potential for automating bureaucratic tasks, surveys, and question-answering systems where an established set of criteria or questions leads to a verdict. It generalizes to any "tedious things you have to fill in" or "stupid processes" in an organization, suggesting a broad application for AI in streamlining compliance and assessment workflows.

These examples collectively paint a picture of AI as a versatile and powerful assistant for security professionals, capable of automating, generating, and transforming, provided the right prompts and model capabilities are employed.

Defensive Implications

▶ Watch: Use case: AI generating specialized security code (BCheck) (36:00)

The rapid proliferation and increasing sophistication of AI introduce a new paradigm for cybersecurity, demanding a proactive and informed defensive posture. Craig Balding's talk outlines several critical implications for defenders.

Firstly, understanding and securing new attack surfaces is paramount. Traditional security models must evolve to account for vulnerabilities like prompt injection, which Balding likens to the "SQL injection of AI." Defenders need to develop strategies to sanitize inputs, detect malicious prompts, and implement robust validation mechanisms for AI interactions. Similarly, model stealing presents an intellectual property risk, requiring AI service providers and organizations to implement countermeasures such as rate limiting, query analysis, and output obfuscation to protect proprietary models.

Secondly, securing the AI data pipeline is as critical as securing the model itself. The talk highlights that "bad data in is bad data out" and that breaches of massive training datasets pose significant risks, including intellectual property theft and exposure of sensitive information. Defenders must ensure robust security controls around data sourcing, collection, cleaning, and storage for AI training data, treating these pipelines with the same rigor as any other critical infrastructure.

Thirdly, the implementation of sandboxing technologies becomes indispensable for AI systems that interact with the environment or execute commands. As AI moves towards "AI in a loop" scenarios where it observes, decides, and acts, sandboxes are crucial to prevent "runaway AI" from making costly mistakes or executing malicious actions, even unintentionally. This calls for a re-evaluation and enhancement of existing sandbox capabilities for AI-specific use cases.

Fourthly, organizations must develop AI-specific security policies. Balding points to the NIST AI Risk Management Framework Playbook as an excellent resource for regulated entities. Policies should, at a minimum, prohibit uploading proprietary or sensitive data to public AI services and provide guidance on responsible AI usage. These policies need to go beyond basic prohibitions to consider the 72 different policy aspects outlined by NIST.

Fifthly, there is an urgent need for security professionals in AI red teaming. Balding notes a significant gap where current AI red teams often lack hacking backgrounds. Defenders with offensive security skills are uniquely positioned to identify novel vulnerabilities, conduct adversarial attacks (like prompt injection), and rigorously test the robustness and alignment of AI models, thereby strengthening their resilience.

Sixthly, monitoring AI outputs, especially when AI monitors AI, is a looming challenge. As businesses move towards automated AI oversight, defenders must establish mechanisms to audit and verify the decisions and actions of these monitoring AIs, ensuring they remain aligned with organizational goals and security policies.

Seventhly, the rise of AI-generated content (deepfakes, fake visuals) complicates authenticity forensics. Defenders will face increasing difficulty in distinguishing genuine content from AI-fabricated material, necessitating investment in advanced detection tools and techniques.

Finally, addressing bias in AI-driven security systems is a complex but crucial task. Biases inherited from training data can lead to unfair or ineffective security decisions, creating a "big can of worms." Defenders must work to identify, understand, and mitigate these biases to ensure equitable and accurate security outcomes. Conversely, Balding also illustrates how AI can be a powerful tool for defenders, automating tedious tasks, generating explanations, transforming unstructured data, and creating custom security tools, provided it's implemented thoughtfully and securely.

Key Takeaways

• AI is a Productivity Game-Changer with New Attack Surfaces: AI offers significant efficiency gains (e.g., 37% faster document writing, 55% faster coding) but introduces novel vulnerabilities like prompt injection ("SQL injection of AI") and model stealing, demanding immediate security attention.
• Comprehensive AI System Security is Essential: Protecting AI goes beyond the model itself; it requires securing the entire data pipeline (from sourcing to processing), the broader system architecture, and implementing sandboxing for AI actions, especially as AI moves towards autonomous operation.
• Security Professionals are Crucial for AI Red Teaming: There's a critical need for individuals with hacking backgrounds to join AI red teams to identify and mitigate vulnerabilities that data scientists alone might miss, strengthening the overall resilience of AI systems.
• Local AI and Data Sovereignty are Emerging Trends: The ability to run AI models locally (e.g., on-device, in-browser via WebGPU) offers enhanced control, privacy, and the potential for domain-specific fine-tuning, reducing reliance on external hosted services for sensitive tasks.
• AI Can Be a Powerful Ally for Defenders: AI can automate tedious tasks (e.g., data transformation, code generation for security tools like Burp Suite BCheck), generate clear explanations of security controls, and assist in cyber intelligence, but its deployment requires careful consideration of data sensitivity and model capabilities.
• Proactive Policy and Oversight are Non-Negotiable: Organizations must develop specific AI security policies, potentially leveraging frameworks like the NIST AI Risk Management Framework, and prepare for a future where AI monitors other AI, necessitating robust human and automated oversight for alignment and risk mitigation.

About the Speaker(s)

Craig Balding is an independent consultant with a keen interest in the applied side of AI security. He emphasizes the practical aspects of implementing and securing AI, drawing from his extensive background in cyber security. Balding is also the author of the "Threat Prompt" newsletter, a publication where he shares five security ideas weekly, demonstrating his commitment to staying current with AI security developments and sharing his insights with the community. He advocates for security professionals to broaden their understanding of adjacent technical domains, believing it makes them more effective in both offensive and defensive roles.