GenAI Application Security: Not Just Prompt Injection

GenAI Application Security: Not Just Prompt Injection

Speaker: Ahmed Abugharbia

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch on YouTube

Reading time: ~7 minutes

TL;DR

GenAI application security is not a completely new field — it is classical security applied to a new architecture. Ahmed Abugharbia, a security researcher and SANS instructor, argues that the key to securing AI systems is first understanding how they actually work, then mapping existing security controls onto the components that are genuinely novel: LLMs, embeddings, vector databases, agents, and their interaction boundaries.

Introduction

By the time Ahmed Abugharbia took the stage at BSidesSF 2025, the audience had already heard multiple talks about AI risks throughout the day. His proposition was a reframing: everything the security industry knows still applies — the details are different, but the high-level approach is not.

Abugharbia holds full-time positions at Fortinet and Cyber Dojo, where he conducts AI security research, and teaches SANS courses SEC 5440 and SEC 545. His central argument: "We cannot secure what we do not understand." Accordingly, the first half of his session is a technical primer on how generative AI actually works — not to explain it away as simple, but to give security practitioners the vocabulary and mental models to attack the right problems.

How Generative AI Actually Works: From Transformers to Agents

▶ Watch: Transformers and LLMs explained (4:00)

The Transformer architecture, introduced by Google in June 2017 in the paper "Attention Is All You Need," underpins all modern large language models. The mechanism is, at its core, predicting the next token in a sequence — the same autocomplete behavior visible in a Google search bar, but generalized to producing any kind of output, including executable instructions. That last point matters: a Transformer-based model is not just a text generator. It is a general-purpose instruction engine that can produce outputs capable of taking actions in other systems.

ChatGPT is just an application — the user-facing layer. The engine behind it is an LLM accessed via API. Abugharbia demonstrates an OpenAI API call in Python, noting that it contains an authentication key (relevant for security) and a JSON body with instructions. This is the integration pattern that is now proliferating across enterprise software: AI components are being added to existing applications alongside their web interfaces, SQL databases, backend APIs, and external service integrations.

He draws out what a modern GenAI application looks like architecturally: an interface layer, traditional application code, one or more LLM models, embedding models, agents (which may be internal or external to the application), and external resources connected via API — search engines, databases, third-party services. The question he poses to the room: raise your hand if you are already scared from a security perspective. Not enough hands, he observes.

Retrieval-Augmented Generation, Fine-Tuning, and Vector Databases

▶ Watch: RAG, fine-tuning, and embeddings (12:00)

LLMs have knowledge cutoffs and limited context for organization-specific data. The two principal techniques for extending their knowledge are fine-tuning and retrieval-augmented generation (RAG).

Fine-tuning takes a pre-trained model — GPT-4, Claude, Llama — and trains it further on domain-specific data to produce outputs in a specific format. The illustrative example: if an agent needs to generate SQL statements and immediately execute them, the output cannot include explanatory prose. It must be a syntactically valid SQL statement and nothing else. Fine-tuning reshapes the model's output distribution to match that requirement.

RAG works differently: a vector database stores recent or proprietary data as embeddings — numeric vectors representing the semantic meaning of text in high-dimensional space. When a query arrives, the application retrieves the most semantically relevant vectors from the database, combines them with the original query, and sends the enriched prompt to the LLM. Abugharbia uses a simplified 2D visualization from ChatGPT: words with similar meanings cluster together in the vector space ("apple," "banana," "pear" cluster; "bike," "car" cluster separately). In practice, these embeddings have hundreds of dimensions.

The implication for security: the vector database is a high-value target. It stores authoritative, often sensitive organizational data. Poisoning it — inserting malicious instructions that get retrieved and included in prompts — is an attack surface distinct from directly attacking the model.

The OWASP LLM Top 10 and Where Attacks Land

▶ Watch: OWASP LLM Top 10 and prompt injection (20:00)

OWASP's Top 10 for LLM applications maps specific attack classes to components in the architecture Abugharbia has described. He highlights the mapping rather than cataloguing every entry:

• Prompt injection affects the model directly, but via multiple entry points: direct user input, indirect input from the vector database (if someone has poisoned stored content), and indirect input from external resources. The example he gives: a GenAI application that retrieves financial information from blogs — does it also process comments? If so, a comment containing injected instructions becomes a prompt injection vector with no special access required.
• Access keys — while not an "AI-specific" risk, they appear prominently in GenAI architectures because integration with external APIs requires authentication credentials embedded somewhere in the application.
• Misinformation / hallucination as a risk category distinct from direct attacks.
• Data poisoning as an attack on knowledge sources rather than the model itself.

His observation about access keys is a direct expression of his central thesis: some of the most serious risks in AI applications are classical security problems dressed in new context.

Supply Chain Attacks Against AI Models

▶ Watch: Model supply chain and pickle exploits (28:00)

The supply chain section is where Abugharbia introduces territory that most AI practitioners have not thought through carefully. Hugging Face hosts tens of thousands of models, uploaded by anyone — Meta, academic researchers, and also unknown individuals. Anyone can push a model to Hugging Face. Anyone can download it. The Hugging Face transformers library makes loading a model as simple as:

That single line downloads and executes a model file from the internet. Models are stored in various formats, including Python's pickle format — a serialized representation of Python objects. Pickle deserialization is well-known to be exploitable: malicious code can be appended to a pickle file, and it will execute when deserialized. This means a model file is functionally equivalent to an executable in terms of the threat it poses, even though it does not have an .exe extension.

Abugharbia has observed thousands of malicious models documented in security research over the past two years. The recommended mitigation is model scan, an open-source tool from Protect AI that parses model files looking for dangerous Python constructs — os.system calls, subprocess invocations, and similar patterns. A model with a backdoor that executes on every inference call is not theoretical; it has already been demonstrated.

MLSecOps: Baking Security Into the Pipeline

▶ Watch: MLSecOps and CI/CD integration (36:00)

Manual scanning is insufficient at scale. Abugharbia's proposed solution draws the direct parallel to DevSecOps: just as security was injected into CI/CD pipelines for code, it must be injected into ML pipelines for models. He demonstrates a GitLab CI/CD pipeline where a scan step is inserted before deployment:

• A Docker container is built containing both application code and the model file.
• Before the container is pushed to production Kubernetes, the scan step runs model scan against the model.
• If a critical finding is detected, the pipeline breaks. The model does not ship.

The same pipeline principle extends beyond model scanning: checksums and signature verification of model files, data validation for training inputs, and audit logging of model provenance. The concept he calls MLSecOps — ML operations with security embedded — mirrors what DevSecOps did for software development. AI/ML researchers are generally not security practitioners; building security into their workflow tooling rather than asking them to add it manually is the only realistic path.

He also raises the option of hosting models locally using tools like Ollama rather than sending queries to third-party APIs, which introduces privacy concerns of its own — particularly relevant when the application processes sensitive organizational data.

Notable Quotes

"We cannot secure what we do not understand. That is why I am spending the first half of this talk on how generative AI actually works."

"Look at the access keys at the top of that slide. Is this AI-related? Not really. But it is heavily used within AI. Back to the argument: we are going to look at the same security controls we have, but adapt them for how applications are changing."

"These models should be treated as executables. They are not executables technically, but they will be loaded into your memory — and they can execute things."

Key Takeaways

• GenAI security is classical security with new architecture. Prompt injection is input validation. Data poisoning is integrity management. Access key exposure is credential hygiene. The controls are familiar; the surfaces are new.
• Prompt injection has multiple entry points. Direct user input is only one. Vector databases, external resources, and comment sections on retrieved web content are all viable injection vectors.
• The model supply chain is an underappreciated attack surface. Hugging Face models are the new untrusted dependencies. Treat them like untrusted executables and scan them before use.
• Pickle deserialization exploits apply to model files. Malicious code can be appended to any pickle-format model. Scanning with tools like model scan before deployment is the baseline defense.
• MLSecOps is the structural solution. Embedding model scanning, signature verification, and data validation into CI/CD pipelines for ML workloads removes the dependency on individual practitioners to remember to check.