Identifying and Assigning AI Model Vulnerabilities

Identifying and Assigning AI Model Vulnerabilities

Speakers: Kyle Kian, Rand Corporation; D Ferguson, Rand Corporation

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=Pz_J1OuXp70

Overview

In an era defined by the rapid proliferation and integration of Artificial Intelligence (AI) across all sectors, understanding and managing its inherent vulnerabilities has become a critical challenge for the cybersecurity community. This talk, presented by Kyle Kian and D Ferguson from the Rand Corporation, delves into the evolving landscape of AI model vulnerabilities and the profound impact AI is having on the traditional vulnerability management ecosystem. They explore whether existing frameworks can adapt to these novel threats or if entirely new paradigms are required.

The presentation dissects the current state of software vulnerability management, highlighting its established components like CVEs, CWE, CVSS, and CVD, and then critically assesses their applicability to AI-specific flaws. Kian and Ferguson introduce new categories of AI vulnerabilities—ranging from conventional software bugs in AI frameworks to complex model behavior flaws and broader ethical/safety issues—and propose emerging frameworks for their disclosure and management. Furthermore, the talk speculates on AI's potential to accelerate both offensive and defensive cybersecurity capabilities, emphasizing the urgent need for adaptive policy solutions and continued research.

This discussion is particularly pertinent as AI systems become increasingly autonomous and integrated into critical infrastructure, decision-making processes, and everyday technologies. The Rand Corporation's ongoing research project aims to address the significant gaps in current vulnerability ecosystems when confronted with AI, offering a roadmap for security professionals, developers, and policymakers to navigate this complex and rapidly changing domain. The insights shared are crucial for anyone involved in securing AI systems or managing vulnerabilities in an AI-driven future.

Background

▶ Watch: Introduction and agenda overview (0:00)

The foundation of modern software vulnerability management rests upon several well-established components, which D Ferguson thoroughly outlined at the beginning of the presentation. These include the Common Vulnerability and Exposure (CVE) standard, which provides unique identifiers for publicly known vulnerabilities; the Common Weakness Enumeration (CWE), a taxonomy of root causes and software/hardware weakness types like buffer overflows or injection flaws; the Common Vulnerability Scoring System (CVSS), which assesses vulnerability severity on a 0-10 scale; and Coordinated Vulnerability Disclosure (CVD), a process for private reporting of bugs to vendors before public disclosure, often facilitated by bug bounty programs. While comprehensive, this ecosystem is primarily designed for traditional software bugs and code errors.

The advent of AI introduces new categories of vulnerabilities and classification challenges that strain the existing framework. Ferguson categorized these into three distinct types:

1. Vulnerabilities in AI software packages: These are conventional software bugs found in AI frameworks or libraries, such as exploitable flaws in TensorFlow or PyTorch. These fit neatly into the existing CVE paradigm and are managed like any other software vulnerability.
2. Model behavior flaws: These represent failures or undesirable behaviors of the AI model itself, rather than bugs in its underlying code. Examples include hallucinations, where an AI generates content that conflicts with its training data or facts, or prompt injection vulnerabilities, where an attacker crafts inputs to make the model behave in an unintended way. Critically, these flaws have no underlying code error to patch, as the issue lies in the model's training or logic.
3. Issues spanning ethics and safety: Categorized as broad AI/ML incidents and failures, these encompass problems like vision model failures in self-driving cars or bias in AI-powered hiring systems. These are not typically security violations being exploited but can have severe, lasting consequences and do not fit neatly into the CVE paradigm.

To illustrate the gaps, the speakers presented a comparative mapping of the traditional software vulnerability ecosystem against the emerging AI/ML vulnerability ecosystem. Key disparities highlighted include:

• Vulnerability Catalogs: The traditional side boasts the comprehensive National Vulnerability Database (NVD) and CISA's KEV catalog. The AI/ML side, however, is missing entirely a central, comprehensive catalog linking unique IDs to detailed AI model flaws.
• Attack Taxonomy: While traditional software benefits from the extensive MITRE ATT&CK framework, the AI/ML space has MITRE ATLAS, which specifically catalogs adversary tactics against AI/ML systems. This shows progress but also a need for specialized frameworks.
• Incident Databases: Established databases exist for traditional incidents (e.g., Privacy Rights Clearinghouse), though they can be fragmented. For AI/ML, dedicated databases like incidentdatabase.ai and the AIC are in development, indicating an emerging focus.
• Bug Bounty Programs/VDB Platforms: The traditional software world has a very mature and established market with platforms like HackerOne and Bug Crowd. In contrast, AI flaw bounty programs are currently emerging and less mature, with platforms like hunter.com and Anthropics' jailbreak challenge representing early efforts.

These comparisons underscore the fundamental challenge: the existing ecosystem, built on the premise of code-level vulnerabilities, struggles to accommodate the unique characteristics of AI model flaws and their broader societal impacts. This necessitates a re-evaluation of how AI vulnerabilities are identified, classified, and disclosed.

Key Findings

▶ Watch: New vulnerabilities and classification challenges with AI models (2:58)

The talk meticulously outlined several key findings regarding AI model vulnerabilities and their implications for cybersecurity:

1. Novel Vulnerability Types: The most significant finding is the emergence of entirely new categories of vulnerabilities specific to AI models. Beyond conventional software bugs in AI frameworks, model behavior flaws (such as hallucinations and prompt injection) and broader ethical and safety issues (like bias or autonomous system failures) represent distinct challenges that do not map directly to the traditional software vulnerability paradigm. These flaws reside in the model's training, data, or logic, rather than in exploitable code errors.

1. Gaps in Existing Ecosystems: The current vulnerability management ecosystem, anchored by CVEs, CWE, and CVSS, is ill-equipped to handle these AI-specific flaws. There is a notable absence of a central, comprehensive catalog for AI model vulnerabilities, akin to the NVD for software. Furthermore, established metrics for severity and root cause analysis struggle to apply meaningfully to issues like AI bias or unintended model behaviors.

1. Emerging Disclosure Frameworks: Recognizing these gaps, new frameworks are being developed to address AI vulnerability disclosure. The Coordinated Flaw Disclosure (CFD) program, proposed by Catel et al., adapts traditional CVD to AI's challenges, incorporating structured reporting, extended model cards, and independent adjudication. Similarly, the Coordinated Disclosure of Dual-Use Capabilities (CDDC), proposed by O'Brien et al., aims to provide an early warning system for dangerous AI capabilities. The rise of AI flaw bounty programs also signifies an industry-led effort to incentivize the reporting of these novel issues.

1. AI's Dual Impact on Offense and Defense: Kyle Kian elaborated on the profound, dual-edged impact of AI on cybersecurity. AI is expected to significantly accelerate both vulnerability discovery (through AI-accelerated code analysis and AI-driven fuzzing) and exploitation (via capability uplift for less sophisticated actors, augmented social engineering, and autonomous agents capable of multi-step attacks). Conversely, AI also offers substantial benefits for defense, enhancing threat detection, real-time monitoring, and alert reduction capabilities.

1. Uncertain Offense-Defense Balance: While AI offers advantages to both sides, the ultimate balance remains uncertain. Most researchers, according to Kian, believe that AI innovations will primarily benefit defenders. However, the potential for AI to democratize access to sophisticated attack techniques for low-skilled actors and the rapid advancement of autonomous agents present significant concerns.

1. Persistent Challenges: Despite the advancements, critical challenges remain in integrating AI into vulnerability management and defense. These include explainability issues in deep learning models (the "black box" problem), the development of adversarial attacks designed to bypass AI detection systems, and inherent failure modes in reinforcement learning (such as reward hacking or specification gaming) that could lead to unexpected vulnerabilities in high-stakes applications.

Technical Deep Dive

▶ Watch: Distinguishing model behavior flaws from traditional code bugs (3:30)

The technical core of the presentation centered on dissecting the unique nature of AI vulnerabilities and the mechanisms by which AI is transforming both offensive and defensive cybersecurity strategies.

D Ferguson meticulously detailed the distinctions between the three AI vulnerability categories. While vulnerabilities in AI software packages are standard code bugs, the model behavior flaws represent a paradigm shift. These are not errors in the code but rather in the learned function of the model itself. For instance, prompt injection involves crafting specific inputs to subvert an AI's intended function, potentially making a chatbot reveal sensitive information or perform unauthorized actions. Hallucinations, common in large language models (LLMs), occur when a model generates plausible but factually incorrect information, which can have significant implications in critical applications where accuracy is paramount. Ethical and safety issues, such as algorithmic bias in hiring systems or critical failures in autonomous driving, highlight that the scope of AI "vulnerabilities" extends beyond traditional security to encompass broader societal and safety concerns.

To address the disclosure of these novel flaws, two key frameworks were introduced:

• The Coordinated Flaw Disclosure (CFD) program, piloted at Defcon 32, is tailored to AI's unique challenges. It advocates for structured reporting, going beyond simple bug reports to include extended model cards. These cards specify the scope, intent, and operational context of the target AI system, providing crucial metadata for understanding and mitigating model-specific flaws. An independent adjudication panel is proposed to mediate disputes between finders and vendors, acknowledging the subjective and complex nature of AI model flaws. A complementary concept, the Common Use Enumeration (CUE), mirrors CWE by attempting to hierarchically organize AI model uses and potential misuses.
• The Coordinated Disclosure of Dual-Use Capabilities (CDDC), proposed in 2024, focuses on AI systems with "dual-use potential"—capabilities that could be beneficial but also unexpectedly dangerous. This framework acts as an early warning system, where finders report risky capabilities to a central clearinghouse, which then alerts appropriate defenders. This proactive approach aims to maximize society's time to prepare defenses, complementing existing model evaluation and red-teaming efforts.

Kyle Kian then shifted to AI's impact on the vulnerability lifecycle, presenting a speculative chart showing a potential 122% jump in CVEs in a single year, similar to a surge seen in 2017. He detailed how AI accelerates vulnerability discovery through:

• AI-accelerated code analysis: Tools like Google's DeepSleep and Code Intelligence's offerings leverage LLMs to scan vast codebases, identifying vulnerabilities much faster than human analysts. The promise is to identify novel vulnerabilities across significantly larger spaces.
• AI-driven fuzzing: Integrating LLM advancements into traditional security tools, such as OSS-Fuzz and CI Fuzz, allows for the development of high-quality test cases, drastically reducing the time required to scrutinize codebases for vulnerabilities. However, Kian cautioned that this is a "double-edged sword," as attackers could use similar techniques to pinpoint weaknesses.

On the offensive side, Kian highlighted AI augmented exploitation:

• Capability uplift: AI tools can empower less sophisticated actors, providing the skills and resources needed to identify vulnerabilities and execute exploits. Models designed to develop malware are already emerging, increasing in sophistication.
• AI-augmented social engineering: AI can generate highly sophisticated and convincing phishing emails on a massive scale, potentially "flooding the zone" with targeted attacks and overwhelming an organization's defenses.
• Autonomous agents: This is perhaps the most significant development. Kian referenced a recent report indicating that AI agents are doubling their capability to complete multi-step tasks every seven months. These agents are progressing towards executing entire kill chains autonomously, from vulnerability discovery to exploitation, in multi-hour projects. The OWASP Top 10 for LLM Agents was recommended as essential reading for understanding these evolving threats.

Conversely, AI also offers substantial advancements for defense:

• Improved threat detection and real-time monitoring: AI integration allows security tools to scan much larger areas and identify subtle deviations in patterns that might indicate malicious activity within massive volumes of data.
• Alert reduction: AI can help security researchers filter out noise, identifying small, critical signals that indicate an impending or ongoing attack, thereby improving efficiency.

However, Kian also emphasized critical challenges for AI-driven defense:

• Explainability challenges: Deep learning models often act as "black boxes," making it difficult to understand why they make certain decisions or identify specific threats. This lack of interpretability is a major concern in high-stakes defense applications. Recent research, such as a paper from Anthropic, is making strides in understanding model behavior.
• Adversarial attacks: Researchers are developing novel techniques to manipulate input data to circumvent AI-based detection systems, posing a constant challenge to the robustness of AI defenses.
• Reinforcement learning failure modes: Many new AI tools use reinforcement learning, which has known failure modes like reward hacking (optimizing for an unintended proxy of the true objective) and specification gaming (exploiting loopholes in the objective function). These could introduce significant, unforeseen vulnerabilities as AI is integrated into more critical processes.

The discussion concluded by framing these insights as part of Rand's ongoing research, focusing on understanding the future capabilities and limitations of AI, the types of vulnerabilities it will introduce across various systems (IoT, SCADA), and the ultimate impact on the offense-defense balance.

Demo / Proof of Concept

▶ Watch: Highlighting gaps in AI vulnerability catalogs and bug bounty (5:45)

The talk focused on the conceptual and theoretical frameworks for understanding and managing AI vulnerabilities, rather than demonstrating a specific exploit or proof of concept. The speakers' presentation primarily served to outline the challenges and potential solutions for integrating AI-specific risks into the broader cybersecurity ecosystem, based on their ongoing research at the Rand Corporation.

Defensive Implications

▶ Watch: Introducing emerging frameworks for AI vulnerability disclosure (7:00)

The insights presented by Kyle Kian and D Ferguson carry significant defensive implications for organizations grappling with the integration of AI and the evolving threat landscape. Defenders must recognize that the traditional vulnerability management paradigm is insufficient for AI and begin to adapt their strategies accordingly.

Firstly, organizations need to re-evaluate and expand their understanding of what constitutes a "vulnerability." This includes acknowledging model behavior flaws like prompt injection and hallucinations, as well as broader ethical and safety issues such as algorithmic bias. Security teams should work closely with AI development teams to understand the model's training data, intended use, and potential failure modes from the earliest stages of development.

Secondly, defenders should monitor and engage with emerging AI vulnerability disclosure frameworks. While the ecosystem is still nascent, participation in or adoption of principles from initiatives like the Coordinated Flaw Disclosure (CFD) program, with its emphasis on extended model cards and structured reporting, can help establish best practices. Supporting AI flaw bounty programs or even launching internal ones for AI models can incentivize responsible disclosure of model-specific weaknesses.

Thirdly, organizations must leverage AI for enhanced defensive capabilities. Investing in AI-powered security tools for improved threat detection, real-time monitoring, and alert reduction is crucial. These tools can help manage the anticipated flood of new vulnerabilities and sophisticated attacks by identifying subtle patterns and reducing the signal-to-noise ratio for security analysts. This includes using AI-accelerated code analysis and AI-driven fuzzing in their own development and security testing pipelines.

Fourthly, defenders need to prepare for AI-augmented offensive capabilities. The potential for AI to democratize sophisticated attacks, scale social engineering efforts, and enable autonomous agents to execute multi-step kill chains means that traditional human-centric security controls and validation processes will be under immense stress. Strengthening defenses against advanced phishing, improving human verification mechanisms, and developing playbooks for responding to highly automated attacks are paramount.

Finally, organizations must invest in research and development to address the inherent challenges of AI security. This includes prioritizing efforts around AI explainability to better understand and trust model decisions in critical defensive applications. Furthermore, developing robust defenses against adversarial attacks that aim to bypass AI detection systems, and understanding and mitigating the unique failure modes of reinforcement learning in AI systems, are essential for building resilient AI defenses. The goal is not just to detect traditional vulnerabilities, but to secure the entire AI lifecycle and ensure the trustworthy operation of AI systems.

Key Takeaways

• AI introduces fundamentally new vulnerability types: Beyond conventional software bugs, AI models exhibit "model behavior flaws" (e.g., prompt injection, hallucinations) and raise "ethical and safety issues" (e.g., bias, autonomous system failures) that traditional vulnerability management is ill-equipped to handle.
• Existing vulnerability frameworks are inadequate for AI: The current ecosystem of CVEs, CWE, and CVSS has significant gaps in classifying, cataloging, and scoring AI model-specific vulnerabilities, necessitating new approaches.
• New disclosure frameworks are emerging: Initiatives like the Coordinated Flaw Disclosure (CFD) and Coordinated Disclosure of Dual-Use Capabilities (CDDC) are being developed to provide structured mechanisms for reporting and managing AI-specific risks.
• AI will accelerate both offense and defense: AI capabilities are projected to significantly boost vulnerability discovery (e.g., AI-driven fuzzing) and exploitation (e.g., autonomous agents, scaled social engineering), while simultaneously enhancing defensive capabilities (e.g., advanced threat detection, alert reduction).
• Challenges in AI defense persist: Key hurdles include AI explainability (the "black box" problem), vulnerability to adversarial attacks, and inherent failure modes in reinforcement learning (e.g., reward hacking) that can introduce unexpected weaknesses.
• Adaptation is crucial for an AI-driven future: Organizations must evolve their vulnerability management strategies to understand and address novel AI threats, invest in AI-powered defensive tools, and contribute to the development of resilient policy solutions and research in AI security.

About the Speaker(s)

The presentation was delivered by Kyle Kian and D Ferguson, both associated with the Rand Corporation. They are actively involved in an ongoing project at Rand that investigates the vulnerability ecosystem and the profound impact of Artificial Intelligence. Sasha Romanowski was also mentioned as a partner on this research but was not present at the conference. Their work focuses on understanding how current vulnerability management frameworks adapt to AI, the emergence of novel AI-specific vulnerabilities, and the broader implications for offense-defense dynamics in cybersecurity.