Navigating the AI Frontier: Investing in AI in the Evolving Cyber...

Navigating the AI Frontier: Investing in AI in the Evolving Cyber...

Speakers: Chenxi Wang

Conference: BSidesSF 2024

YouTube: https://www.youtube.com/watch?v=YQEe_QpmCzM

Overview

Dr. Chenxi Wang, a distinguished computer scientist, former academic, and now a prominent investor, delivered a keynote address at BSidesSF 2024, focusing on the transformative impact of Artificial Intelligence (AI) on the cybersecurity landscape. Her talk, "Navigating the AI Frontier: Investing in AI in the Evolving Cyber Landscape," provided a unique perspective, bridging the gap between deep technological understanding and the realities of market trends and investment opportunities. Dr. Wang highlighted that while AI has been a concept for decades, the recent boom, particularly since November 2023 with the advent of Large Language Models (LLMs) like ChatGPT, marks a significant shift, moving from theoretical exploration to widespread experimentation and potential pervasive deployment.

The core of Dr. Wang's presentation revolved around how AI is fundamentally reshaping both the offensive and defensive paradigms in cybersecurity. She argued that traditional assumptions about data advantage and proprietary knowledge are eroding, necessitating a complete re-evaluation of security strategies, application security (AppSec), and even the very nature of software development. From an investor's standpoint, she also explored the burgeoning market for AI-driven security solutions and the inherent risks and opportunities in this rapidly evolving sector, emphasizing the need for robust, innovative approaches that go beyond mere "thin layers" on existing LLMs.

This article delves into Dr. Wang's insights, exploring the technical shifts, the implications for defenders, and the strategic considerations for navigating this new AI-powered frontier. It aims to provide a comprehensive understanding of the challenges and opportunities that AI presents to the cybersecurity community, drawing directly from the detailed observations and predictions shared during her keynote.

Background

▶ Watch: High inference/training costs hindering pervasive AI deployment (05:00)

The rapid ascent of AI, particularly generative AI and LLMs, has been nothing short of phenomenal. Dr. Wang underscored this by citing compelling statistics: ChatGPT achieved one million users in just five days, a feat that took Instagram two months, Spotify five months, Facebook ten months, and Netflix three and a half years. This unprecedented adoption rate signifies a fundamental shift in how technology is consumed and integrated into daily life and business operations. Estimates suggest that approximately 200 million companies worldwide are currently experimenting with LLMs, indicating a massive wave of interest and exploration.

Despite this widespread experimentation, Dr. Wang noted that pervasive deployment of AI in critical business processes is not yet a reality. A significant hurdle remains the high cost associated with inferences and training AI models. Many companies developing AI products in lab environments are often surprised by the expense when moving to production, leading to a cost-benefit analysis where the added value of AI does not yet justify the operational costs and associated privacy and security concerns. This economic reality suggests that while we are in an "experimental stage," a "pervasive deployment stage" requires further advancements in cost efficiency.

Dr. Wang, drawing from her background as a computer scientist with a PhD and experience teaching at Carnegie Mellon, highlighted a profound transformation in the nature of software engineering itself. She posited that software engineering is evolving into model engineering, where the low-level mechanics of coding (e.g., syntax, assignment vs. equality operators) will increasingly be subsumed by AI-driven code generation. The focus for engineers will shift towards "telling the model what to do," testing its behavior, and understanding the cost mechanics of running it as an application. This paradigm shift necessitates a re-evaluation of how future generations are educated in programming, moving away from traditional syntax-heavy instruction towards leveraging pre-built models and understanding their application within an AI stack.

Furthermore, the speaker emphasized that the traditional dynamics of security, often reliant on a defender's proprietary knowledge and superior data access, are rapidly changing. The "data inequality" between defenders and attackers is diminishing, as AI provides new avenues for attackers to gather intelligence and craft highly targeted assaults. This erosion of traditional defensive advantages forms a critical backdrop for understanding the evolving cyber landscape.

Key Findings

▶ Watch: Generic LLMs can replace point products by processing security data (12:00)

Dr. Wang's keynote unveiled several critical findings regarding the intersection of AI and cybersecurity, offering both challenges and opportunities:

• Transformation of Software Engineering: The field is shifting from traditional software engineering to model engineering. AI will automate low-level coding mechanics, requiring engineers to focus on model orchestration, testing, and cost management rather than syntax.
• Diminishing Data Inequality: AI is leveling the playing field between attackers and defenders. Attackers can now leverage AI to process vast amounts of publicly available data, convert disparate security information into consumable formats, and infer architectural weaknesses or product usage within target organizations, eroding the defender's traditional advantage of proprietary environmental knowledge.
• Evolution of Data and Application Security: Traditional data security (discovery, classification, access management) and AppSec are no longer sufficient. New requirements include securing the data used for training and fine-tuning AI models, ensuring data quality (e.g., lack of bias, malicious intent), and performing sensitivity and prediction tests on models. AppSec is becoming model security.
• AI Models as a Threat Vector: The widespread adoption of a few dominant LLMs (e.g., OpenAI, Anthropic, Llama) creates a monoculture risk. A vulnerability in one of these foundational models could lead to a "SolarWinds of models," impacting numerous organizations built on top of them.
• Open Source Models vs. Open Source Software: The term "open source" for AI models is often misleading. While code might be available, the crucial training data and model weights are frequently proprietary, meaning users lack full transparency and control compared to traditional open-source software.
• AI as a Significant Opportunity for Security: Despite the threats, AI presents immense opportunities for defenders. It can automate tasks like root cause analysis, boost productivity, and process large volumes of security data more effectively. The potential to automate 80-90% of the current $100 billion spent on security professional services represents a massive market shift.
• The Power of Retrieval Augmented Generation (RAG): RAG architectures, which combine public LLMs with private, internal data sources (e.g., endpoint logs, xdr logs, network logs, email data, policies) via vector databases, offer a powerful way to leverage AI for specific organizational security insights, potentially replacing numerous point security products.

Technical Deep Dive

▶ Watch: DevOps evolving to 'Micro Models' for continuous deployment (23:00)

The technical implications of AI in cybersecurity are profound, touching upon software development, data handling, and defensive strategies. Dr. Wang elaborated on several key areas:

1. Software Engineering to Model Engineering:

The shift is from writing explicit code with precise syntax to model engineering. This means leveraging AI for code generation, where the AI handles the low-level mechanics. The engineer's role transforms into defining the model's objectives, testing its outputs, and understanding the operational costs of running the model in production. This involves grappling with concepts like inference costs and training costs, which are currently significant barriers to pervasive AI deployment. The focus moves from "how to code" to "how to leverage a model" and "what is important for the application to be manifested in this AI stack."

2. Diminishing Data Inequality and Attack Surface Expansion:

AI significantly alters the information asymmetry between attackers and defenders. Dr. Wang described a "toy demo" where an individual could pull information from various infrastructure components and security products, convert this data into flat text files, generate embeddings, and then use a lightly modified third-party LLM against a vector database to ask questions like, "What is the biggest gap in the security architecture overall?" This capability allows for sophisticated analysis of an organization's security posture from potentially external or semi-external data sources.

The ability to ask questions like, "Does Bank of America use product X?" and "Does Bank of America use product X in such a way?" and receive accurate answers, enables attackers to direct their efforts much more precisely. This is further exacerbated by the emerging data economy in underground markets, where data brokers are selling datasets specifically for training and fine-tuning attack-oriented models, rather than just raw credit card or identity data. The speaker noted her personal experience of asking specific, non-widely known questions about herself to ChatGPT and receiving accurate answers, highlighting the LLM's vast and often surprising knowledge base. This capability, when harnessed by nation-states or sophisticated criminal groups training "attack brains" akin to GPT-5, poses an unprecedented threat.

3. Evolving Data Security and Application Security (AppSec) to Model Security:

Traditional data security focuses on data discovery, classification, access management, and permission management. However, with AI, the "boundary of data" expands. It now includes:

• Training data: The data used to initially train an AI model.
• Fine-tuning data: Data used to adapt a pre-trained model to specific tasks.
• Inference data: Data fed to the model during its operational use.

New security requirements emerge:

• Data provenance and integrity: How do we ensure the right data is being used? Is it biased? Does it contain malicious intent?
• Testing: Developing methods for sensitivity testing (how a model's predictions change with slight data variations) and prediction testing (evaluating accuracy against specific data categories). Dr. Wang suggested A/B testing models trained with and without specific datasets to understand their impact.

AppSec, traditionally focused on static analysis, dynamic analysis, and penetration testing of application code, must now encompass model testing. This includes:

• Training data validation: Understanding the different datasets, their characteristics (e.g., dark vs. bright images, distorted images for image recognition models), and their impact on model behavior.
• Model supply chain security: Identifying and securing all chained models, especially when open-source models might rely on other backend models.
• Robustness testing: Testing against specific samples, cuts, and use cases, including prompt injections and adversarial data training.
• Red teaming: Evolving pen testing to red team against AI models and data.

The speaker highlighted that many enterprise products, including security products, are already incorporating LLMs on the backend, often without explicit customer knowledge. This raises concerns about the robustness and security testing of these embedded models and the data they process.

4. DevOps to ModelOps:

The shift from monolithic applications to microservices revolutionized testing, deployment, and identity management. Dr. Wang predicted a similar transformation with AI, where microservices will evolve into micro-models communicating via APIs and data exchange channels. This introduces new challenges for ModelOps:

• Continuous deployment of models: How to deploy new models into an existing architecture while ensuring the collective decision-making of the entire model chain remains robust.
• Drifting and hallucinating models: What happens when one micro-model in a chain starts to drift in its predictions or hallucinates? How does this impact the overall application's reliability and security?

5. AI Model as a Threat Vector:

The increasing reliance on a few popular third-party LLM APIs (OpenAI, Anthropic, Llama) creates a monoculture risk. If one of these widely deployed foundational models has a security vulnerability, it could be exploited to compromise numerous organizations simultaneously, akin to a "SolarWinds of models." This necessitates a focus on the security of these foundational models themselves.

6. Open Source Models: A Nuanced View:

Dr. Wang cautioned that "open source" for AI models is not equivalent to open-source software. While the code might be available, the critical training data and model weights are often proprietary. This lack of transparency means users cannot fully understand or audit the model's behavior, biases, or potential vulnerabilities in the same way they can with traditional open-source software.

7. AI for Security: Retrieval Augmented Generation (RAG):

One of the most promising technical applications of AI for security is Retrieval Augmented Generation (RAG). Public LLMs are trained on vast internet data but lack access to an organization's internal, proprietary information (e.g., legal documents, policy documents, emails). RAG addresses this by:

• Internal Vector Database: An organization builds an internal vector database populated with its own data (e.g., endpoint logs, xdr logs, network logs, VPC flow data, email data, policies). This data is "chunked," indexed, and converted into embeddings.
• Query Augmentation: When a user submits a query, a logic layer first sends it to the internal vector database.
• Similarity Search: The vector database performs a similarity search to retrieve relevant internal data.
• Prompt Construction: This retrieved internal data is then packaged together with the original user query to form an augmented prompt.
• LLM Inference: The augmented prompt is sent to a public LLM, which then answers the question based on both its general knowledge and the specific private data provided.

This architecture allows LLMs to provide highly relevant and context-aware answers to security questions, such as identifying the "largest gaping hole in the security architecture." A key technical challenge highlighted is that while vector databases excel with unstructured data, much of security data (e.g., log data) is highly structured. Innovating ways to effectively transform and chunk structured security data for consumption by LLMs via vector databases is an active area of research and investment.

Demo / Proof of Concept

▶ Watch: AI models as a threat vector (monoculture risk) (26:00)

Dr. Wang illustrated her points with several compelling examples and demonstrations, both hypothetical and real-world:

1. Security Architecture Gap Analysis (Toy Demo): A "toy demo" was described where a person set up a stage environment to pull information from various infrastructure components and security products. This disparate data was then converted into flat text data files, embeddings were generated, and these were stored in a vector database. A lightly modified third-party LLM was then used to query this database, asking questions like, "What is the biggest gap in the security architecture overall?" The AI was able to provide insightful answers, demonstrating how a generic LLM, when fed specific organizational data, can perform sophisticated security analysis that traditionally required proprietary analysis algorithms from point products.

1. Targeted Exploitation via Public Information: Dr. Wang presented a hypothetical but plausible scenario involving a real CVE affecting a specific version of Ubuntu with an exploit in the wild. The scenario involved an attacker knowing that a particular organization does not use a specific product (Product Y) that protects against this exploit, or that they use a different product (Product X) that is ineffective in that environment. The speaker noted that information about "who uses what product is actually pretty easy to get these days," and it's not a stretch to imagine LLMs having access to such data. This capability would allow an attacker to "very easily... unleash this exploit on this company" with high precision. She also referenced her own experience asking ChatGPT obscure, non-widely known questions about herself and receiving accurate answers, underscoring the vast data LLMs can access and process.

1. AI in Customer Service (Insurance Company Case Study): A concrete example from a large US insurance company (potentially number one or two in the US) demonstrated the business benefits and security concerns of AI deployment. During the annual health insurance open enrollment period, this company traditionally hired 25,000 temporary human agents to answer subscriber questions.

• Human Agent Performance: Human agents achieved an average accuracy of 66.7% when answering questions based on tens of thousands of policy documents. They also consistently received low empathy scores due to the nature of calls (subscribers often being in a bad mood).
• AI Agent Performance: When AI agents were introduced into the mix, their accuracy soared to 99.x%. Furthermore, AI agents, not getting tired or mad, achieved higher empathy scores.
• Business Impact: The company planned to reduce human agents from 25,000 to 12,000 in the current year, supplementing with AI, and potentially reducing to zero in the future.
• Security Concern: The company's primary fear was that competitors could send human agents to work for them, use the AI agent tools, and through a series of "interesting questions," easily extract the underlying business logic (e.g., how policies are underwritten or monetized) from the AI model. This highlights the need to obscure AI models and protect prompts to prevent the exfiltration of critical business intelligence.

These examples collectively illustrate both the immense potential of AI to automate and enhance various functions, and the novel security challenges that arise from its deployment.

Defensive Implications

▶ Watch: Challenges with structured security data in vector databases for RAG (38:00)

The insights shared by Dr. Wang necessitate a significant re-evaluation of defensive strategies in cybersecurity. Defenders must adapt to a landscape where traditional advantages are eroding and new threat vectors are emerging.

1. Rethink Data Advantage and Proprietary Knowledge: Defenders can no longer solely rely on having proprietary knowledge of their environments or more data than attackers. AI enables attackers to synthesize information from various sources, potentially inferring internal architectural details or product usage. Security teams must assume that external entities can gain sophisticated insights into their infrastructure and adjust their defenses accordingly, focusing on resilience and proactive threat hunting rather than just obscurity.

1. Embrace Model Security as a Core AppSec Discipline: The scope of application security must expand dramatically to include model security. This means:

• Securing the AI Supply Chain: Understanding and validating the provenance of all data used for training and fine-tuning models, as well as the integrity of chained models (the "model supply chain").
• Robust Testing for AI Models: Implementing rigorous testing methodologies that go beyond traditional code analysis. This includes sensitivity testing, prediction testing, and specific tests for prompt injections and adversarial data training. Red teaming efforts should be specifically geared towards exploiting AI models and their underlying data.
• Bias and Malicious Intent Detection: Developing mechanisms to detect and mitigate bias or malicious intent embedded within training data or model outputs.

1. Address the Monoculture Risk of Foundational LLMs: The widespread adoption of a few dominant LLMs (e.g., OpenAI, Anthropic, Llama) creates a significant single point of failure. Defenders must:

• Diversify or Isolate: Explore strategies to diversify their reliance on foundational models or implement robust isolation and monitoring for critical applications built on these LLMs.
• Demand Transparency and Security from Providers: Pressure LLM providers for greater transparency regarding their models' security posture, training data, and vulnerability management processes.
• Prepare for "SolarWinds of Models": Develop incident response plans specifically for supply chain attacks targeting widely used AI models.

1. Leverage AI for Defensive Automation and Efficiency: While AI poses threats, it also offers powerful defensive capabilities. Organizations should actively explore and invest in AI for security applications such as:

• Automated Root Cause Analysis: Using AI to rapidly process vast amounts of log and telemetry data to identify the root causes of security incidents.
• Threat Detection and Response: Enhancing existing security tools (SIEM, XDR) with AI-driven analytics for more accurate and timely threat detection.
• Productivity Boosting: Automating repetitive tasks for security analysts, freeing them to focus on more complex strategic challenges.
• Rethink Security Product Stacks with RAG: Consider how Retrieval Augmented Generation (RAG) architectures, integrating internal security data with LLMs, could potentially replace or consolidate numerous point security products. This requires a strategic assessment of existing tools and a willingness to innovate.

1. Protect Business Logic Embedded in AI Models: As AI models become repositories of critical business logic (as seen in the insurance company example), organizations must implement measures to prevent the extraction of this intellectual property. This could involve techniques to obscure model internals, secure prompt engineering, and monitor for suspicious query patterns.

1. Prepare for ModelOps: The evolution from DevOps to ModelOps requires new approaches to continuous integration/continuous deployment (CI/CD) for AI models. Defenders need to understand how to secure architectures composed of interacting "micro-models," monitor for model drift or hallucination, and ensure the collective decision-making remains secure and reliable.

Key Takeaways

• AI is fundamentally reshaping cybersecurity: Traditional defensive advantages based on data inequality and proprietary knowledge are diminishing, requiring a paradigm shift in security strategies.
• Software engineering is transforming into model engineering: The focus for developers will shift from low-level coding mechanics to orchestrating, testing, and managing AI models, with significant implications for skill sets and education.
• Model security is the new AppSec: Securing AI models involves new considerations like training data integrity, model supply chain security, prompt injection robustness, and adversarial data testing, extending beyond traditional application code analysis.
• AI models introduce new threat vectors: The widespread adoption of a few foundational LLMs creates a monoculture risk, potentially leading to large-scale supply chain attacks akin to a "SolarWinds of models."
• Retrieval Augmented Generation (RAG) offers significant defensive opportunities: By combining public LLMs with private, internal security data via vector databases, RAG can provide powerful, context-aware insights, potentially consolidating or replacing numerous point security products.
• The economic impact of AI in security is massive: AI has the potential to automate 80-90% of the $100 billion spent annually on security professional services, presenting a huge market opportunity for innovative solutions.

About the Speaker(s)

Dr. Chenxi Wang is a highly accomplished computer scientist, investor, and thought leader in the cybersecurity industry. She holds a PhD in computer science and began her career as an academic, teaching computer science at Carnegie Mellon University. Transitioning to industry, Dr. Wang gained extensive experience in product strategy, market research, and executive roles at both large and small companies. Currently, she is an investor, straddling the business and technology worlds, which provides her with a unique perspective on the viability and impact of emerging technologies. Dr. Wang has a strong background in application security, having written an obfuscation compiler for her PhD thesis and served on the board of OWASP. She is the founder of Rain Capital, an early-stage investment firm focused on cyber and AI.