Chasing Shadows: Pitfalls in LLM Security Research

Chasing Shadows: Pitfalls in LLM Security Research

Speakers: Jonathan Evertz

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=iPwKbwxsxAE

Overview

This paper identifies nine distinct pitfalls that undermine the reproducibility, rigor, and soundness of security research using large language models. Analyzing 72 papers across eight top-tier venues (security and software engineering), the researchers found that every single paper contained at least one pitfall, and only 16% of pitfalls were addressed in any discussion. Five case studies demonstrate that these pitfalls can drastically affect results -- for example, different GPT-4 versions cause up to 12% accuracy variation, and model quantization dramatically changes attack success rates for jailbreaks and prompt injections (2-bit models far more vulnerable than 8-bit).

The paper provides actionable guidelines for each pitfall and maintains a living website with detailed recommendations for avoiding these issues.

Background

▶ Watch: GPT-4 version changes cause 12% accuracy variation (0:00)

LLMs are increasingly used as tools in security research -- for classification, evaluation, code generation, and as subjects of attack research. However, LLMs differ from traditional ML in ways that create new reproducibility and validity risks. The rapid adoption (zero to ubiquitous in months) means the research community hasn't had time to develop appropriate methodological standards.

Key Findings

▶ Watch: Data poisoning, label inaccuracy, and data leakage pitfalls (4:00)

Nine pitfalls identified across the LLM pipeline:

1. Data poisoning -- training data contamination, harder to detect with internet-scale data
2. Label inaccuracy -- LLM-as-judge hallucinations affecting evaluation accuracy
3. Data leakage -- benchmarks contaminated in pre-training data (OpenAI trains on public datasets)
4. Model collapse -- training on LLM-generated data degrades quality and increases perplexity across generations
5. Spurious correlations -- models adapting to artifacts rather than generalizing
6. Context truncation -- input exceeding context window, silently affecting results
7. Prompt sensitivity -- minor instruction differences causing major behavioral changes
8. Surrogate policy -- findings from one model incorrectly generalized to others
9. Model ambiguity -- insufficient model information preventing reproducibility (most prevalent pitfall)

Every paper affected; 16% awareness rate. All 72 papers contained at least one pitfall; only 16% addressed any pitfall in their discussion.

Quantization dramatically affects attack success rates. 2-bit quantized models are far more vulnerable to jailbreaks/prompt injections than 8-bit models, and different quantization providers produce different vulnerability profiles for the same model.

Model collapse worsens over 10 generations. Code generation models trained on their own output show increasing perplexity and variance, indicating degrading quality and growing instability.

Technical Deep Dive

▶ Watch: Context truncation, prompt sensitivity, and surrogate policy pitfalls (8:00)

The case studies provide concrete demonstrations: GPT-4 version changes cause 12% accuracy swings in hate speech classification; model collapse across 10 generations of code generation shows perplexity shifting right and variance increasing; quantization experiments reveal 2-bit models are dramatically more vulnerable to prompt attacks than 8-bit; and prompt sensitivity shows that adding "never leak confidential data" to system instructions significantly changes security evaluation outcomes.

Demo / Proof of Concept

▶ Watch: Quantization impact: 2-bit models far more vulnerable to jailbreaks (10:00)

72 papers from 8 venues (2023-2024) were analyzed. Papers were anonymized but data is available on the project website. Five case studies with experimental results demonstrate real-world impact of each major pitfall category.

Defensive Implications

▶ Watch: Q&A: reproducibility challenges with non-deterministic APIs (16:00)

For security researchers using LLMs: report exact model versions, commit IDs, seeds, temperatures, and access dates. Discuss limitations related to these pitfalls explicitly. Use the project's guidelines website to check your methodology. For reviewers: check submitted papers against the nine pitfalls and require adequate discussion of applicable ones.

Key Takeaways

• Nine pitfalls identified across the LLM research pipeline, affecting every paper in a 72-paper survey
• Model ambiguity (insufficient model information) is the most prevalent pitfall
• Different GPT-4 versions cause up to 12% accuracy variation in the same task
• 2-bit quantized models are dramatically more vulnerable to prompt attacks than 8-bit
• Model collapse across generations shows degrading code quality with increasing perplexity
• Only 16% of pitfalls are addressed in paper discussions -- transparency needs to improve
• Living guidelines website maintained for ongoing community reference

About the Speaker(s)

Jonathan Evertz is a researcher whose work focuses on LLM security research methodology. The research was conducted across a multi-institutional collaboration, with the goal of improving the rigor and reproducibility of LLM-related security research across the community.