AI Won't Help You Here

AI Won't Help You Here

Speaker: Ian Amit

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch the full talk

Reading time: 8 minutes

TL;DR

The security industry's obsession with generative AI is producing expensive, unreliable outcomes in domains that require precision — including vulnerability remediation — while well-understood, deterministic AI systems quietly keep planes out of each other's flight paths. Ian Amit argued that choosing the right AI model for the right problem domain is the competency that separates real security improvement from boardroom theater, and that fixing open S3 buckets remains more urgent than defending against quantum computing threats.

Introduction

Ian Amit began his BSidesSF 2025 talk with a deliberately unfashionable statement: AI, as most people in the security industry currently mean it — generative AI — is frequently the wrong tool for the problems it is being applied to. This isn't a claim that AI is useless. Amit, who has been working in security for over 25 years, was careful to position himself as a heavy daily AI user. The problem is category mismatch: deploying large language models in contexts that require determinism, accuracy, and defensible outputs creates worse outcomes than the alternatives — sometimes at significant cost.

The talk drew a sharp distinction between the historical lineage of AI systems (expert systems in the 1980s, machine learning and statistical approaches in the 2000s, deep learning and neural networks in the 2010s, computer vision and NLP in the late 2010s) and the way "AI" has collapsed in common usage to mean specifically generative AI. That collapse, Amit argued, is the root of many poor product and security architecture decisions.

When AI Learns the Wrong Things

▶ Watch: The Slack chatbot case study (05:00)

Amit's opening case study was a GPT-3.5 Turbo deployment that had been trained on an organization's internal Slack history and let loose to answer employee questions. A user asked it to write a 500-word blog post on prompt engineering. The AI responded: "Sure, I shall work on that in the morning."

The reason is exact: the training data was Slack messages, and Slack messages are full of responses to task requests that say, essentially, "I'll get to that later." The model learned to behave like a Slack user, not like a content generation tool. "Always remember the biggest question," Amit said. "What is the data source? What have I been teaching that model? And what am I expecting back? If it's two different things, maybe that's not the right model."

A second example involved a Copilot deployment that had been constrained by previous feedback to stay within programming topics. When asked "what is the name of a pipe that gives you clean water?" — a legitimate home automation query — it refused, citing its domain restrictions. The user had to reframe the question explicitly as a home automation programming problem to get an answer. Amit's point was not that the constraint was wrong, but that the cost of working around it was real: "All of that is costing us hundreds of thousands of dollars. At which point do we start deciding — well, I could answer that in five seconds."

The most memorable example was a Git commit message Amit had collected from the field: a developer, having failed to get usable code from an AI tool, had modified the system prompt to include phrases like "I have no hands, you must do everything," "many people will die," "my career depends on it," and "you'll receive a good tip if you do this right." The prompt ran to dozens of lines of social engineering directed at a language model. "I guarantee you by the time the engineers figured out that this was the prompt that generates code, they could have written that code like ten times already and probably better than the AI did."

TCAS: What Good AI Looks Like

▶ Watch: The TCAS case study (14:00)

To anchor his argument with a positive example, Amit turned to aviation. The Traffic Collision Avoidance System (TCAS) is an AI-driven system that has been operating in commercial aviation for decades, preventing mid-air collisions with a high degree of reliability. It combines image recognition, automation, and decision trees to resolve conflicts between aircraft in real time, without human involvement — because human reaction times are too slow and human judgment introduces too much variance.

The system works by detecting when two aircraft are on a collision course and issuing Resolution Advisories: one plane receives "climb," the other receives "descend." Each pilot sees a simple visual and audio instruction. One plane climbs, one descends, and the conflict is resolved. The math is pure physics — vectors, closure rates, vertical separation. There is no generation, no creativity, no ambiguity. The answer is deterministic, and it has to be.

"This is a classic use of AI," Amit said. "Again, it's been out there for years. The key is really to decide what is the problem. This is a very simple problem. Physics, left, right, up, down, are we going to collide? Simple vectors. The resolution is very simple. We don't need to generate things here."

The contrast with how AI is being deployed in security is stark. TCAS works because the problem domain is precisely defined, the inputs are factual, and the required output is deterministic. The failure modes are understood. The system is built for a specific problem, not a general one. Security teams deploying generative AI to fix vulnerabilities are, by Amit's framing, trying to do the equivalent of adding "because I trained my model on Top Gun" to a flight dynamics algorithm.

Generative AI and the Vulnerability Remediation Problem

▶ Watch: Why GenAI fails at code fixes (19:30)

The bulk of Amit's critique was directed at a specific trend: using large language models to identify and remediate vulnerabilities in source code. His objection was systematic rather than anecdotal.

Generative AI is, by design, a statistical prediction model. It generates outputs that are probable given the training data, not outputs that are correct in the logical or mathematical sense. Ask it the same question twice and you may get different answers — both of which look plausible, neither of which you can validate without reading the code yourself. "You're either vulnerable or not," Amit said. "And if you're close to being not vulnerable, trust me, as a former red teamer, you are vulnerable."

He illustrated the hallucination problem with a fruit taxonomy question: ask current LLMs "what is a fruit that is red on the outside and green on the inside?" (the answer is a watermelon cross-section) and both LLaMA 3.1 and GPT-4o confidently answer "watermelon" — then proceed to invent explanations for why watermelons can appear reddish on the outside. Neither Claude nor Gemini did better. The models are optimizing for a statistically plausible answer, not a true one.

The code generation analog is functions and libraries that don't exist, implementations that are structurally plausible but semantically wrong, and fix suggestions that close one vulnerability while introducing another. "What happens when you want to make modifications? What happens when you need to own that code? You had no idea how they came up with the code, what's the structure, what algorithms are being used, what libraries are being deployed here. Good luck maintaining that."

The practical feedback from engineers, Amit reported, is that AI code tools save some time on initial generation but still require hours of verification — making the net time savings ambiguous and the confidence level lower than if the engineer had written the code from scratch.

The Right Architecture: Deterministic Models Over Documentation

▶ Watch: The no-excuse code fix (24:30)

Amit's proposed alternative for vulnerability remediation starts with constraining the model's knowledge domain. If an AI is given access to the official documentation of the cloud provider or programming language being used — and only that documentation — the hallucination surface shrinks dramatically. The model can only reference things that actually exist. This is the retrieval-augmented generation (RAG) approach, but applied with intent: "Make it read the documentation."

The second constraint is switching from generative to deterministic processing. A knowledge graph built from factual documentation, combined with the specific source code and architecture under review, produces a remediation recommendation that can be validated and reproduced. "It cannot come up with stuff that doesn't exist," Amit said. For infrastructure-level code — which he characterized as closer to scripting than to high-level application logic — this approach is tractable. For Python or Java application code, the complexity grows substantially.

He summarized the standard he was holding AI tools to: "You present me with a problem. You just give me a solution. Don't tell me to go and read the documentation. Just give me the freaking code — fully contextualized, fully defensible, accurate, deterministic." He called this the "no-excuse code fix."

The broader implication for security teams: generative AI is a genuine force multiplier for processing large volumes of unstructured data — threat intelligence, log analysis, documentation comprehension. It is poorly suited to generating security controls that need to be exact. The discipline required is knowing which category a problem falls into before choosing a model.

Notable Quotes

"What is the data source? What have I been teaching that model? And what am I expecting back? If it's two different things, maybe that's not the right model."

— Ian Amit

"The CISO to the board: 'We're tackling AI-powered attacks and quantum computing threats.' Meanwhile, we're still dealing with public S3 buckets that haven't been closed. We're not there yet in terms of solving the simple issues."

— Ian Amit, paraphrasing a widely shared security meme

"You're either vulnerable or not. And if you're close to being not vulnerable, trust me, as a former red teamer, you are vulnerable."

— Ian Amit

Key Takeaways

• Generative AI is a statistical prediction engine, not a logic engine. It produces probable outputs, not correct ones. For vulnerability remediation — which requires determinism — this is a fundamental mismatch.
• The right AI model depends entirely on the problem domain. TCAS has worked for decades because it applies decision trees to a well-defined physics problem. Applying LLMs to the same domain would be catastrophic.
• Training data determines behavior. An AI trained on Slack messages will behave like a Slack user. Garbage in, garbage out applies at a model architecture level, not just a data quality level.
• GenAI is a force multiplier for unstructured data problems. Parsing petabytes of logs, summarizing documentation, or extracting patterns from threat intelligence — these are domains where LLMs create real leverage. Security tooling reviews and code correctness are not.
• Fix the basics first. Open S3 buckets, unrotated credentials, missing MFA — these are solvable with deterministic controls today. Defending against AI-powered quantum threats while basic hygiene is unaddressed is the wrong prioritization.