Operationalizing AIBOMs: Extending Vulnerability Management to AI Models and Datasets

Operationalizing AIBOMs: Extending Vulnerability Management to AI Models and Datasets

Speakers: Alexandra Selldorff, Manifest Cyber; Ugur Koc, Manifest Cyber

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=k58hZqnNUbM

Overview

In an era where Artificial Intelligence (AI) has become a ubiquitous production dependency across virtually all digital products and workflows, the security landscape for these systems remains remarkably immature. Traditional vulnerability management (VM) programs, while highly sophisticated for software and hardware, are ill-equipped to address the unique and complex risks inherent in modern AI. This talk, delivered by Alexandra Selldorff and Ugur Koc from Manifest Cyber at VulnCon, addresses this critical gap by proposing a structured framework for AI governance centered around the concept of an AI Bill of Materials (AIBOM).

The speakers highlight that current AI security reviews are often manual, inconsistent, and burdensome, largely due to the "black box" nature of AI assets. This lack of transparency and standardization impedes scalable security assessments and policy enforcement. The AIBOM is presented as the foundational solution, offering a machine-readable, standardized "ingredients label" that encompasses not only the traditional software dependencies but also the AI-specific artifacts like model weights, training datasets, and configuration files. This innovation is crucial for extending robust vulnerability management practices to the burgeoning domain of AI, enabling automated risk assessment and continuous monitoring throughout the machine learning lifecycle.

Background

▶ Watch: Introduction to AI vulnerability management gap (0:00)

The evolution of technology has seen the security industry develop highly mature vulnerability management programs for conventional software and hardware components. However, the rapid integration of AI systems into critical infrastructure and consumer products has exposed a significant blind spot. AI assets are inherently complex, comprising a diverse array of components that extend far beyond typical software binaries. These include model weights (often massive files representing the learned parameters), datasets (used for training, fine-tuning, evaluation), text processing components like tokenizers, adapter layers for fine-tuned models, and standard software dependencies (e.g., PyTorch, TensorFlow) along with crucial configuration files that dictate model behavior.

These diverse artifacts become relevant at different stages of the machine learning lifecycle, from initial training and fine-tuning, through evaluation and testing, to deployment in production and subsequent continuous monitoring. Managing the security posture across this complex, multi-stage pipeline with traditional tools is nearly impossible. While model cards exist as documents providing an overview of a model's specifications (parameters, architecture, task, datasets, licenses), they fall short for security purposes. Model cards are typically unstructured, often incomplete, and lack standardization. This makes them unsuitable for automated policy enforcement, a critical requirement for any scalable AI governance program. The absence of a structured, machine-readable inventory for AI components is the fundamental problem that the AIBOM aims to solve, paving the way for proactive and automated AI security.

Key Findings

▶ Watch: Visualizing AI-BOM structure and content (6:00)

The central finding and contribution of this talk is the AI Bill of Materials (AIBOM) as the essential, structured, and machine-readable solution for AI governance. The speakers unequivocally state that an AIBOM is, in essence, an S-bomb (Software Bill of Materials), but "glorified" to specifically include AI artifacts. This extension allows for the application of established S-bomb principles to the unique domain of AI, overcoming the limitations of unstructured model cards.

The AIBOM, as demonstrated by the speakers, provides a comprehensive, standardized inventory of an AI system's constituents. Key categories of information within an AIBOM include:

1. Model Identity and Lineage: This encompasses the model's name (e.g., Q13G Jan 8B), version, supplier, country of origin, licensing information, and crucially, its ancestor model if it's a fine-tuned version (e.g., parent model Q138B). This lineage information is vital for tracing provenance and understanding potential inherited risks.
2. Structured Model Card: Unlike its unstructured predecessor, this AIBOM component provides machine-readable details about the model's intended task, architecture, performance metrics (e.g., accuracy for classification models), intended users, use cases, ethical considerations, and technical limitations. This structured data enables automated policy checks against declared model behaviors.
3. Dependencies and Files: This section lists all necessary software libraries (e.g., transformers library) required to load and use the model, complete with package identifiers (such as Package URLs or PURLs). PURLs are critical for automated vulnerability matching. It also enumerates all files shipped with the model, including configuration files, model weights, and specifically calls out the potential presence of executable software or sensitive information like secrets, which can pose significant risks.
4. Dataset Information: Details about the datasets used in the model's lifecycle (e.g., evaluation data), including references to research papers that introduced these datasets. This structured data is paramount for assessing data-centric risks such as bias, poisoning, or privacy leakage.

By providing this standardized and machine-readable inventory, the AIBOM enables automated policy enforcement, allowing organizations to define configurable policies that check various attributes like model lineage, supplier, freshness/staleness, and dataset declarations. This dramatically improves the scalability and consistency of AI security reviews, moving away from burdensome manual processes and allowing for the proactive discovery and management of vulnerabilities not only in the AI artifacts themselves but also in their underlying software dependencies.

Technical Deep Dive

▶ Watch: Unique data and governance risks in AI (10:00)

The core technical contribution of the AIBOM lies in its ability to enable robust policy enforcement against a novel set of risks unique to AI systems. The speakers meticulously categorize these risks, explaining why traditional vulnerability management programs are ill-equipped to handle them:

1. Data-Centric Risks:

• Tampered/Poisoned Datasets: Malicious actors can inject poisoned data during training, causing models to misbehave in production.
• Bias and Failures: Datasets unrepresentative of diverse populations can lead to models treating certain groups unfairly, posing significant social and ethical risks, particularly in sensitive applications like law enforcement or hiring.
• Privacy Leakages: Models trained on sensitive personal information (e.g., health data) can inadvertently leak this data, violating privacy regulations.

1. Governance-Heavy Risks:

• Misalignments and License Conflicts: Using a model outside its intended use cases or in violation of its license terms can lead to legal and ethical repercussions.
• Outdated Models: Models can suffer from data drift (where real-world data diverges from training data) or hallucination (making up information), leading to unreliable performance.
• Regulation Compliance: AI models must adhere to regional regulations (e.g., GDPR in the EU prohibits using demographic data for hiring), and non-compliance can result in severe penalties.

To address these, the talk proposes concrete policy examples that can be enforced using AIBOMs:

• Model Staleness/Freshness: Policies to prevent using models that are too old (risk of data drift, hallucination) or too new (risk of undiscovered zero-day vulnerabilities, similar to software).
• Blocklists: Organizations can block models from specific countries, suppliers, or those known to be problematic due to biased training data or other issues.
• Software Files: Prohibiting models from shipping with executable scripts, which could act as backdoors to gain system access.
• License Compliance: Enforcing the use of models with licenses approved by the organization.
• Training Data Declaration: Requiring models to declare their training data for auditability and to address concerns about data quality, bias, or privacy.

The technical implementation of AIBOM generation and policy evaluation was demonstrated using both OWASP's open-source AIBOM generator and a custom CLI tool developed by Manifest Cyber. While OWASP's tool provides a foundational AIBOM, the custom CLI tool enriches it with more granular information such as ancestor models, detailed file lists, comprehensive software dependencies, archived research papers, and contributor information. This richer AIBOM, structured in formats like SPDX or CycloneDX, allows for precise vulnerability matching using PURLs for software dependencies, enabling the identification of CVEs (e.g., numerous CVEs linked to a specific version of the transformers library). The demo also highlighted geopolitical risks, where affiliations of software dependency contributors with foreign agencies could be flagged for organizations like Google.

Looking ahead, the speakers outlined crucial future work:

• Contextualizing AI: Moving beyond static analysis to understand how models are used in production. This involves analyzing source code to infer use cases (e.g., an HR software using an AI model) and dynamically inspecting network traffic or API calls to third-party AI vendors to deduce usage context.
• Deeper Dynamic Analysis: Beyond static AIBOM analysis, dynamically testing models for sensitive information leakage, unauthorized executable calls, and mitigating risks outlined in the OWASP LLM Top 10 (e.g., prompt injection, sensitive information disclosure, poisoning, prompt leakage, misinformation).
• Agentic AI and MCP Tools: Investigating supply chain vulnerabilities in sophisticated agentic AI systems and Multi-Agent Coordination Protocol (MCP) tools, focusing on risks like identity and privilege abuse or exploitation by the models themselves.

These advancements underscore the need for a multi-faceted approach, combining static AIBOM analysis with dynamic testing and contextual awareness to build a truly resilient AI security posture.

Demo / Proof of Concept

▶ Watch: Demo: AI-BOM generation and policy evaluation (16:50)

The talk included a compelling demonstration of an AI governance workflow, illustrating how AIBOMs can be operationalized in practice. The workflow begins with the acquisition of an open-weight model, typically from platforms like Hugging Face.

1. AIBOM Generation: The first step in the demo involved generating an AIBOM for a chosen model. Initially, an AIBOM was created using OWASP's open-source AI bomb generator, producing a relatively concise S-bomb (e.g., 158 lines) that included basic model identity, manufacturer, and a structured model card. To obtain richer metadata crucial for comprehensive policy evaluation, a custom CLI tool developed by Manifest Cyber was then used. This tool generated a more detailed AIBOM, explicitly listing the model's ancestor, individual files shipped with the model, all software dependencies (with versions and PURLs), related datasets, archive papers, and even contributors.

1. Policy and Risk Scanning: With the detailed AIBOM in hand, the system proceeded to perform policy and risk scanning. The demonstration showed an example where a model was determined to be "high risk" with "medium security findings." Specific flags included an "unrecognized supplier" and the model "ships with software," indicating potential concerns.

1. Detailed Findings and Vulnerability Identification: The system allowed for a deep dive into the findings. It displayed the extracted dataset information and, crucially, the software dependencies, highlighting the transformers library and its version. A critical part of the demo was showing that this specific version of transformers had "quite a few CVEs" mapped to it, underscoring the importance of including traditional software dependency scanning within the AIBOM framework. Furthermore, the demo identified "foreign risk" associated with the package, stemming from contributors of its software dependencies having affiliations with foreign agencies, a concern for organizations like Google.

1. Evaluation and Continuous Monitoring: The demonstration concluded by emphasizing that the output from policy and risk scanning informs the decision to approve or deny a model for use. Approved models are then stored in a model dataset and inventory, which is subject to continuous monitoring. This ensures that as new vulnerabilities are discovered, policies change, or model characteristics evolve (e.g., staleness), the model is re-evaluated, potentially leading to its removal from the approved list or necessitating mitigation. The workflow also highlighted the importance of tracking model deployment to quickly identify where a flagged model is in use, enabling prompt remediation.

This proof of concept effectively showcased how a structured AIBOM, combined with automated scanning and continuous monitoring, provides a scalable and robust mechanism for managing the security and governance of AI models throughout their lifecycle.

Defensive Implications

▶ Watch: Q&A: AI-BOM includes software dependencies (24:30)

The insights from this talk offer several critical defensive implications for organizations grappling with AI security:

• Adopt AIBOMs as a Standard: Organizations must move beyond unstructured model cards and actively generate and utilize AIBOMs for all AI models, whether open-source or internally developed. This is the foundational step to extend existing vulnerability management programs effectively to AI systems.
• Automate AI Governance: Leverage the machine-readable nature of AIBOMs to automate policy enforcement. This means defining clear, configurable policies (e.g., on model lineage, supplier, freshness, dataset declaration, allowed licenses) and integrating AIBOM scanning into CI/CD pipelines for AI models, moving away from manual, inconsistent reviews.
• Prioritize Continuous Monitoring: AI models, their dependencies, and the threat landscape are dynamic. Establish a continuous monitoring workflow for AIBOMs to re-evaluate models against policies over time, flagging new risks (e.g., newly discovered CVEs in dependencies, model staleness, changes in regulations) and ensuring timely remediation.
• Address AI-Specific Risks Holistically: Implement policies and controls specifically designed to mitigate data-centric risks (poisoning, bias, privacy leakage) and governance-heavy risks (license conflicts, regulatory compliance like GDPR). The AIBOM provides the necessary data to assess these risks proactively.
• Scrutinize Software Dependencies: Do not overlook the traditional software components required to run AI models. A comprehensive AIBOM must include these dependencies, enabling standard vulnerability scanning for CVEs in libraries like PyTorch or TensorFlow, just as with any other software.
• Control Executable Files and Secrets: Enforce policies that prevent AI models from shipping with executable scripts or embedded secrets. These can serve as backdoors or lead to sensitive information disclosure if exploited.
• Track Model Lineage and Datasets: Require explicit declaration of model lineage (ancestor models) and training/evaluation datasets. This transparency is crucial for understanding provenance, assessing inherited risks, and auditing data quality, bias, and privacy implications.
• Consider Geopolitical Supply Chain Risks: Evaluate the origins of models, their suppliers, and the contributors to their underlying software dependencies. As demonstrated, affiliations with foreign agencies can introduce unique supply chain risks that organizations, particularly those in critical sectors, need to consider.
• Prepare for Advanced AI Security: Begin exploring methods for contextualizing AI usage (understanding how a model is used in production) and planning for dynamic analysis to detect runtime risks like sensitive data leakage or unauthorized system calls, preparing for the challenges posed by the OWASP LLM Top 10 and agentic AI systems.

Key Takeaways

• AIBOMs are the Foundation for AI Security: An AI Bill of Materials (AIBOM) is an essential, structured, and machine-readable inventory that extends the concept of an S-bomb to include AI-specific artifacts like model weights, datasets, and configuration files, alongside traditional software dependencies.
• Traditional VM Fails for AI: Existing vulnerability management programs are inadequate for AI systems due to their unique risks, including data poisoning, bias, privacy leakage, license conflicts, and regulatory compliance challenges.
• Automated Policy Enforcement is Key: AIBOMs enable scalable and consistent automated policy enforcement across the entire machine learning lifecycle, checking attributes like model lineage, supplier, freshness, and dataset declarations.
• Comprehensive Coverage is Crucial: Effective AIBOMs must provide a holistic view, encompassing both AI-specific components (models, datasets) and their underlying software dependencies, facilitating traditional CVE scanning and identifying broader supply chain risks.
• Continuous Monitoring is Non-Negotiable: Due to the dynamic nature of AI models and their environment, continuous monitoring of AIBOMs and re-evaluation against policies are critical to maintain a robust security posture.
• Future Focus on Dynamic Analysis and Context: Future advancements in AI security will concentrate on dynamic analysis for runtime risks, contextualizing AI usage within deployed systems, and addressing emerging threats in agentic AI environments.

About the Speaker(s)

Alexandra Selldorff is a speaker from Manifest Cyber. Her expertise contributes to the development and operationalization of AI governance frameworks, as demonstrated by her co-presentation on AIBOMs.

Ugur Koc is also a speaker from Manifest Cyber. He identifies himself as an "AI governance professional by training," indicating a specialized background in the principles and practices of governing AI systems. His work focuses on addressing the security and compliance challenges inherent in AI models and datasets.