AI Systems Are Software Systems

AI Systems Are Software Systems

Speakers: Jonathan Spring, CISA

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=Gm-jwyYwiLU

Overview

In an era saturated with discussions about artificial intelligence, Jonathan Spring of CISA (Cybersecurity and Infrastructure Security Agency) delivered a pivotal talk at VulnCon titled "AI Systems Are Software Systems." Spring's central thesis is both straightforward and profoundly important: AI systems are not an exception to the fundamental principles of software security. This presentation serves as a critical call to action for the cybersecurity community, urging them to extend existing vulnerability management frameworks, secure development practices, and risk assessment methodologies to the rapidly evolving domain of AI.

The talk clarifies a crucial distinction: it focuses on the cybersecurity of AI software systems rather than the use of AI to enhance cybersecurity operations. Spring argues that the often-perceived novelty and complexity of AI should not exempt it from established security rigor. Instead, he posits that many "new" AI security challenges are merely existing software security problems rephrased in AI jargon, requiring only minor adaptations—roughly a "10% change, not a 90% change"—to current practices. The implications are far-reaching, demanding a proactive integration of AI engineering into the broader cybersecurity ecosystem to prevent the proliferation of insecure AI deployments.

Background

▶ Watch: AI systems are software systems: core thesis (0:00)

The premise of Spring's talk stems from a recognized gap in how AI systems are often perceived and secured compared to traditional software. Despite being built upon code, models, and data, AI systems frequently bypass the stringent vulnerability management and secure development lifecycle processes that are standard for other software. This oversight is partly due to the rapid pace of AI innovation, a lack of cybersecurity expertise among many AI engineers, and a widespread misconception that AI inherently operates outside the established norms of software security.

Spring highlights that this isn't an entirely new phenomenon. Historically, other specialized software domains, such as medical devices, Operational Technology (OT) systems, and cloud systems, initially presented unique security considerations. Over time, however, they were successfully integrated into comprehensive cybersecurity frameworks. The speaker draws parallels to a decade or two ago when medical device engineers were surprised to learn they were "writing software" and had to conform to security update policies, which eventually led to the FDA changing rules to allow security fixes without full recertification. This historical context suggests that the AI community, while facing its own set of specific challenges, is on a similar trajectory towards formalizing its security posture.

Existing foundational work already provides a strong basis for securing AI. Spring references collaborative efforts like the UK and international partners' guidelines for AI system development, as well as the NIST Secure Software Development Framework (SSDF) AI profile. These documents, he notes, largely map existing secure software development principles onto AI contexts, often simply translating common security concepts into AI-specific terminology. For instance, the general cybersecurity concept of protecting against adversarial input is directly analogous to the AI community's concern with prompt injection. Similarly, CISA's Cross-Sector Cybersecurity Performance Goals (CPGs), which outline baseline cybersecurity practices for critical infrastructure, are found to be largely applicable to AI systems with minimal adjustment in language. The problem, therefore, isn't a lack of guidance, but rather a lack of adoption and integration by the AI community, often due to unfamiliarity with established cybersecurity paradigms.

Key Findings

▶ Watch: Differentiating AI software quality from security issues (11:00)

Jonathan Spring's core message is that the cybersecurity community already possesses the tools and frameworks to secure AI systems. The primary "key finding" is that AI systems are, at their heart, software systems, and this fundamental truth dictates that they must be treated as such within the realm of cybersecurity. This perspective directly challenges the notion that AI requires an entirely new, bespoke set of security principles and practices. Instead, Spring asserts that existing vulnerability management, secure by design, and risk management frameworks are largely sufficient, requiring only "about a 10% change, not a 90% change" to accommodate AI's nuances.

A critical discovery highlighted in the talk is the prevalent conflation of software quality problems with software security vulnerabilities within the AI community. Spring emphasizes that issues like a model providing a factually incorrect answer (e.g., misclassifying a cat as a dog) are typically software quality problems, not security vulnerabilities. Historically, organizations have separate teams—such as a PSIRT (Product Security Incident Response Team) for security and a knowledge management team for factual errors—to handle these distinct types of issues. He argues that this separation must be maintained for AI systems, even if the same tools might identify both types of problems.

Furthermore, Spring identifies that many perceived "new" AI security challenges have direct, well-documented parallels in traditional software engineering. Concepts like naming conventions, versioning schemes, stochastic behavior, and supply chain attacks are not unique to AI. For example, the difficulty in reproducing a sporadic AI model error is likened to managing race conditions in traditional software, a problem that has existed since the early days of computing and has established mitigation strategies and CVSS metrics. This reframing is crucial for bridging the communication gap and enabling the cybersecurity community to apply its extensive knowledge to AI.

Technical Deep Dive

▶ Watch: LLM non-determinism as an engineering choice (25:00)

The technical core of Spring's argument revolves around applying established software security concepts and frameworks to AI systems, often by translating AI-specific terminology into familiar cybersecurity language. He methodically dissects various aspects of vulnerability management, demonstrating their applicability to AI.

Firstly, Spring reiterates CISA's clear stance on vulnerability definitions, adhering to congressional mandates. He emphasizes that AI systems are federal information systems and are therefore subject to existing policies. This includes mandatory actions like patching Known Exploited Vulnerabilities (KEVs). If an AI system has a KEV, it doesn't get a pass; it must be patched. Similarly, organizations deploying AI systems must have a vulnerability disclosure policy, appropriately scoped to security vulnerabilities, not factual inaccuracies. A newer requirement (CISA 2602, January) mandates that if a device is no longer receiving supported security updates, it must be decommissioned, forcing AI system developers to plan for long-term security update support. Crucially, Spring advocates for demanding secure by design principles in AI system acquisition and development, stressing that security features are not free or accidental but require upfront investment.

For building and deploying AI systems, Spring points to existing guidance. The NIST SSDF AI profile and international guidelines provide blueprints for secure AI development. He highlights CISA's Cross-Sector Cybersecurity Performance Goals (CPGs), noting that this document largely maps existing CPGs to AI-specific jargon. For instance, prompt injection in AI is conceptually equivalent to protecting against adversarial input in traditional systems – both are about screening bad inputs. The process of Testing, Evaluation, Validation, and Verification (TEV), while testing slightly different properties for AI, remains an established software development process, not a new concept.

Identifiers and Versioning are areas where AI systems are perceived to have unique challenges. Spring dismisses this, stating that naming is hard for all software. He endorses the use of Package URLs (PURLs) for identifying models, noting that they were integrated into CVE records six months prior. While AI model versioning can be "weirder" (non-linear improvements), he questions if semantic versioning is perfectly applied in traditional software anyway, suggesting the challenge isn't entirely novel.

The application of CVSS (Common Vulnerability Scoring System) to AI-related topics is discussed. Spring notes that while it can be challenging, the Stakeholder-Specific Vulnerability Categorization framework is designed to accommodate new decision trees for different system types. He also clarifies the distinction between software security and quality problems, referencing a recent CVSS FAQ item. He criticizes the common practice of AI teams conflating these, urging them to route factual errors to quality teams and security issues to cybersecurity teams, just as is done for other software.

Asset and Configuration Management are identified as perennial challenges exacerbated by AI. Spring confronts the common complaint that Software Bills of Materials (SBOMs) are impossible to do at scale by citing Deutsche Bahn, the German train company, which processes 500,000 SBOMs daily across its entire IT/OT infrastructure using mostly open-source tools. This example serves as undeniable "proof of existence" that SBOMs are achievable at scale, even for complex systems, and that AI doesn't inherently make it harder. He analogizes models to libraries and datasets to configuration files, emphasizing that tracking these components is crucial, just as it is for any other software dependency. AI systems, he argues, merely bring existing deficiencies in asset and configuration management to the forefront.

Spring delves into specific CVE assignment rules to illustrate their direct applicability:

• CVE Operational Rule 4.2.7: A vulnerability does not require a fix for a CVE to be issued. This directly addresses the AI engineer's common complaint that retraining a model for a fix might take nine months, which is not a valid reason to withhold a CVE.
• CVE Operational Rule 5.1.11.1: SaaS vulnerabilities are encouraged to receive CVEs. This applies even if the AI system is a fully hosted SaaS offering where the customer has no control.
• Race Conditions: Spring likens the stochastic and hard-to-reproduce problems in AI systems to race conditions in traditional software. He points out that LLMs are often non-deterministic by design (using pseudo-random number generators without exposed seeds), making them less problematic than true race conditions if the seed were exposed.
• CVE Operational Rule 4.1.4: Insecure default configurations should be determined to be vulnerabilities. This rule is crucial for many AI-related security findings.
• CVE Operational Rule 4.1.7: Detection bypasses should not get CVEs. This rule directly addresses model misclassifications. If a model misclassifies malware, it's typically a detection bypass, not a CVE-assignable security issue, aligning with the intuition of many AI engineers.
• Malware in AI Models: For cases where models themselves are malware (e.g., typo-squatted, trojanized), the existing rules for supply chain attacks on package managers apply. If a legitimate model is compromised and turned into malware, it gets a CVE. If it's a generally malicious package from the start, it typically does not.

Finally, Spring clarifies the concept of integrity by differentiating between system integrity (unauthorized change to system behavior, often a CVE), data integrity (unauthorized change to data in transit or storage), and correctness integrity (accuracy and consistency of information, typically a quality issue). He suggests that CVSS likely refers to system integrity, and making this distinction clear will help integrate AI engineers into the security community.

Demo / Proof of Concept

▶ Watch: Why AI misclassifications are often not CVEs (28:00)

While Jonathan Spring's talk did not feature a live technical demonstration, he provided a compelling real-world example that serves as a proof of concept for applying traditional vulnerability management to AI systems: a CVE involving the Keras lambda layer.

Spring recounted his involvement in a vulnerability where the Keras lambda layer, a component used in deep learning frameworks, was documented as running in a safe, sandboxed mode. However, in reality, it was not. This meant that users downloading random model files from the internet, which the system was designed to process, could inadvertently execute arbitrary code. The core issue was an insecure default configuration that misled users about the safety of their operations.

Despite documentation warning against running untrusted code, Spring argued that if a program is designed to do a specific action, and that action leads to a security vulnerability under default settings, it is indeed a vulnerability. He emphasized that the default settings should reflect actual safety. This situation directly invoked CVE Operational Rule 4.1.4, which states that insecure default configurations should be determined to be vulnerabilities. The Keras team initially expressed reluctance about assigning a CVE, but it was ultimately assigned and subsequently fixed. This case clearly illustrates how an AI-specific software vulnerability, despite initial resistance from the developers, was successfully processed and addressed using established CVE rules and the broader cybersecurity community's norms. It underscores Spring's central argument: AI systems are software systems, and their vulnerabilities are managed like any other.

Defensive Implications

▶ Watch: Call to action: Engage your AI engineering teams (40:00)

The implications of Spring's talk for defenders are clear and actionable. The primary directive is to treat AI systems as software systems without exception, integrating them fully into existing cybersecurity frameworks and processes.

1. Apply Existing Policies and Frameworks: Defenders must ensure that all AI systems within their scope adhere to established organizational security policies. This includes mandatory patching of Known Exploited Vulnerabilities (KEVs), establishing clear vulnerability disclosure policies for AI products, and implementing plans for decommissioning AI systems that no longer receive security updates, as per CISA's recent directive (CISA 2602). Organizations should also leverage existing guidance like the NIST SSDF AI profile and map their AI security efforts to the Cross-Sector Cybersecurity Performance Goals (CPGs).

1. Demand Secure by Design: Proactive security is paramount. When acquiring, purchasing, or building AI systems, organizations must embed security requirements from the outset. This means putting "secure by design" principles into procurement contracts and development requirements, recognizing that upfront investment in security is more cost-effective than remediation after a compromise.

1. Implement Robust Asset and Configuration Management (including SBOMs): Organizations must bolster their asset and configuration management practices to include AI models and datasets. This involves generating and utilizing Software Bills of Materials (SBOMs) for all AI components, treating models as libraries and datasets as configuration files. The example of Deutsche Bahn processing 500,000 SBOMs daily demonstrates that this is achievable at scale and is not an insurmountable task unique to AI.

1. Distinguish Software Quality from Security Vulnerabilities: Defenders need to educate AI engineers and product teams on the critical difference between a model making a factual error (a quality issue) and a model exhibiting a security vulnerability (e.g., allowing unauthorized access or control). PSIRTs should only handle security issues, ensuring that quality problems are routed to appropriate teams, preventing the overwhelming of security resources with non-security incidents.

1. Proactive Engagement with AI Engineers: Perhaps the most crucial defensive implication is the call for product security teams (PSIRTs) and CISOs to actively engage with AI engineers. This involves reaching out, building relationships, and translating cybersecurity concepts into language that resonates with AI development teams. The goal is to integrate security into the entire AI development pipeline, rather than treating it as an afterthought or a separate hurdle. This means helping AI engineers understand how CVE rules apply to their work and why existing security practices are beneficial.

1. Embed Security into Development Pipelines: Recognizing that not all AI engineers will be cybersecurity experts, defenders should advocate for and implement automated security checks and controls directly into CI/CD (Continuous Integration/Continuous Delivery) pipelines for AI development. This "encoding" of security ensures that vulnerabilities are caught early and that engineers cannot bypass necessary security measures, preventing "vulnerability fountains" caused by rapid, unchecked code generation with AI.

1. Strategic Sandboxing for AI Agents: For AI agents that interact with untrusted inputs or generate code, defenders should learn from historical lessons in securing web browsers and operating systems. Spring explicitly references the vulnerabilities of early Internet Explorer being part of the OS kernel. This highlights the need for robust sandboxing and isolation mechanisms for AI components that handle potentially malicious or unpredictable data, preventing them from compromising the host system.

Key Takeaways

• AI Systems are Software Systems: AI is not an exception to established cybersecurity principles; existing vulnerability management, secure by design, and risk management frameworks apply directly.
• Adapt, Don't Reinvent: Securing AI requires minor adaptations (a "10% change") to existing cybersecurity practices, not a complete overhaul or entirely new methodologies.
• Separate Quality from Security: Clearly distinguish between software quality issues (e.g., factual inaccuracies, misclassifications) and true security vulnerabilities, routing them to appropriate teams for remediation.
• Engage and Educate AI Engineers: Product security teams must proactively reach out to AI development teams, translate security concepts into their context, and integrate secure development practices into their CI/CD pipelines.
• Embrace SBOMs and Asset Management: Implement robust asset and configuration management, including Software Bills of Materials (SBOMs) for AI models and datasets, recognizing that this is achievable at scale.
• Learn from History: Leverage lessons learned from securing other complex software systems like web browsers and medical devices to design secure architectures for AI, especially concerning untrusted inputs and code execution.

About the Speaker(s)

Jonathan Spring is an expert from the Cybersecurity and Infrastructure Security Agency (CISA), a component of the United States Department of Homeland Security. His role at CISA involves shaping federal cybersecurity policy and guidance, particularly concerning vulnerability management and emerging technologies. Throughout his talk, Spring demonstrates a deep understanding of CVE operational rules, NIST frameworks, and the practical challenges of integrating new technologies into established security paradigms. His insights are grounded in extensive experience with governmental cybersecurity initiatives and collaborations with international partners.