Disrupting the Model: Abusing MLOps Platforms to Compromise ML Models and Enterprise Data Lakes

Disrupting the Model: Abusing MLOps Platforms to Compromise ML Models and Enterprise Data Lakes

Speakers: Brett Hawkins (IBM X-Force Red); Chris Thompson (IBM X-Force Red)

Conference: ShmooCon

YouTube: https://www.youtube.com/watch?v=Cqz1ovrhnsU

Overview

In an era where nearly every organization is rapidly integrating Artificial Intelligence (AI) into its operations, the security of the underlying Machine Learning Operations (MLOps) platforms remains a critically overlooked domain. This talk, presented by IBM X-Force Red leaders Chris Thompson and Brett Hawkins, sheds light on this pervasive blind spot, highlighting the significant gap in offensive and defensive research, tooling, and guidance for securing the environments used to train, tune, deploy, and manage AI models. The speakers introduce MLLo Kit, a novel open-source toolkit designed to simulate attacks against these platforms, thereby empowering organizations to test their defenses and build robust security postures.

The core premise of the research is that while "AI red teaming" often focuses on model-centric attacks like prompt injection or adversarial evasion, a much broader and more critical attack surface exists within the entire MLOps ecosystem. This includes the interconnected infrastructure, APIs, agents, and Identity and Access Management (IAM) systems that underpin AI development and deployment. By demonstrating practical methods for abusing the REST APIs of popular commercial MLOps platforms, Thompson and Hawkins reveal how adversaries can achieve objectives ranging from data and model extraction to lateral movement and privilege escalation within an enterprise network. Their work serves as a vital call to action, urging the security community to expand its focus beyond the model itself to encompass the comprehensive security of the MLOps pipeline.

Background

▶ Watch: Introduction to abusing MLOps platforms (3:20)

MLOps is defined as the practice of deploying and maintaining machine learning models in a secure, efficient, and reliable manner. It encompasses a consistent, automated process for the entire ML lifecycle, from data collection and model development to testing, deployment, management, and monitoring. This lifecycle typically involves five key phases: design (data collection, sanitization, tagging), develop (model building and training, leveraging GPU banks), test (model validation), deploy (making the model available in production), and manage (ongoing monitoring and maintenance).

The rapid adoption of AI has led many organizations to leverage commercial MLOps platforms like Azure ML, Amazon SageMaker, and Google Cloud Vertex AI. While these platforms streamline AI development, their security implications are often underestimated. The speakers emphasize that the industry's current focus on "AI red teaming" is often too narrow, primarily concentrating on the models themselves (e.g., direct/indirect prompt injection, model/weight extraction, adversarial evasion). However, a red teamer's perspective extends to the wider ML ecosystem, including training environments, tuning environments, deployment orchestrators, plugins, APIs, agents, and their integrations with IAM and underlying infrastructure (virtual machines, containers).

This expanded attack surface presents a lucrative target for various threat actors. MLOps platforms often have direct access to or sit adjacent to enterprise data lakes and Crown Jewel data, and they are deeply integrated with core backend systems, virtual infrastructure, container infrastructure, and IAM. A compromise of these systems can quickly lead to initial access, expanded access, and lateral movement within a target organization. A critical concern highlighted is the widespread lack of preparedness among incident response (IR) teams. Most commercial platforms do not enable diagnostic logging or audit logging by default, and even when enabled, logs are rarely shipped to a Security Information and Event Management (SIEM) system for custom rule development. This leaves IR teams ill-equipped to detect sophisticated attacks targeting MLOps infrastructure or backdoored models.

Prior research in this space, notably by Adrien Wood and Mary on supply chain attacks involving backdoored models on platforms like Hugging Face, focused on the risk data scientists incur by downloading and evaluating potentially malicious models. Academic work by Natesh and Florine also contributed to understanding model vulnerabilities. However, the IBM X-Force Red research distinguishes itself by taking a comprehensive red teamer's perspective, focusing on the broader ecosystem and the actionable attack paths available through REST API abuse of MLOps platforms, a domain previously lacking public tooling and detailed talks.

Key Findings

▶ Watch: Overview of talk agenda and research topics (4:40)

The central contribution of this research is the development and public release of MLLo Kit, an open-source tool designed to simulate attacks against MLOps platforms. This toolkit directly addresses a significant gap in the security landscape, as there was previously zero tooling available to automate and streamline offensive operations against these environments. MLLo Kit enables security professionals to perform reconnaissance, extract sensitive data, and exfiltrate trained models by abusing the platforms' REST APIs, often leveraging stolen API keys or access tokens.

The research's key findings underscore several critical points:

1. Lack of Offensive Tooling: Before MLLo Kit, security teams and red teamers had no public, purpose-built tools to simulate attacks against commercial MLOps platforms. This left organizations vulnerable and unable to effectively test their defenses.
2. REST API Abuse as a Primary Vector: The talk demonstrates that the REST APIs of MLOps platforms, designed for programmatic interaction, are ripe for abuse. By understanding the underlying API calls, attackers can perform a wide range of malicious actions, including model and data extraction, without direct interaction with the platform's user interface.
3. Absence of Public Detection Guidance: Concurrently with the lack of offensive tooling, there has been a notable absence of public detection rules and threat hunting guidance specifically tailored for MLOps environments. This leaves organizations ill-prepared to identify and respond to compromises.
4. The "Wider Ecosystem" is the Real Target: While model-specific attacks receive significant attention, the research emphasizes that the most impactful objectives for sophisticated threat actors lie in compromising the broader MLOps infrastructure. This provides access to critical data, enables lateral movement, and offers avenues for privilege escalation, far beyond just manipulating a single model's output.
5. Default Insecurity: A recurring theme across multiple commercial MLOps platforms is that essential security features, such as diagnostic logging and audit logging, are often not enabled by default. This necessitates manual configuration and integration with SIEMs, a step frequently missed by organizations rushing to adopt AI.

By bringing an offensive security expert's perspective to MLOps, Thompson and Hawkins have not only provided a valuable tool but also ignited a crucial conversation about the need for a holistic approach to AI security, moving beyond model-centric views to secure the entire operational pipeline.

Technical Deep Dive

▶ Watch: Redefining AI red teaming: beyond just models (6:40)

The attack surface of MLOps platforms is vast, extending beyond the models themselves to encompass the entire lifecycle and integrated infrastructure. The speakers delineate various threat actor motivations and corresponding attack objectives, many of which can be achieved through the abuse of MLOps platform APIs.

Threat Actor Motivations and Attack Objectives:

• Resource Exhaustion/DDoS: Basic threat actors might use public tooling to send numerous queries, consuming tokens and resources, leading to significant financial costs for organizations.
• Supply Chain Attacks & Backdoored Models: More sophisticated adversaries might seed backdoored models on public repositories like Hugging Face. Data scientists, constantly downloading and evaluating models, risk introducing malicious code. This could lead to initial access for initial access brokers or wider supply chain compromises.
• Targeting Model Extensions & Chaining Vulnerabilities: Advanced actors might focus on bypassing new security measures like SafeTensors or chaining vulnerabilities to achieve containerization breakouts.
• Financial Fraud: A major bank, for instance, tasked IBM X-Force Red with targeting their AI-based fraud analytics platform. The goal was to steal quantitative trading algorithms and compromise the platform to modify models or rule sets, allowing malicious trades to go undetected.
• Nation-State Espionage: State-sponsored actors, particularly those heavily invested in AI, aim to steal models (e.g., military AI for satellite imagery, SAR image recognition) or backdoor them to render them useless, disrupting an adversary's defensive capabilities (e.g., misidentifying a warship as a cargo ship).
• Compromising Security AI Tools: Attackers could poison training data for security helper AI tools (e.g., Microsoft Security Copilot, EDR platforms) to make them ignore specific Indicators of Compromise (IOCs), granting a huge advantage in future campaigns.
• Data Extraction: Targeting the raw source data or training datasets can yield sensitive information (financial, healthcare, intellectual property like source code). Ransomware groups or industrial spies would find this data invaluable.
• Model Extraction: Stealing a trained model can provide insights into its inner workings, training data, and weights. This could be used for competitive advantage (e.g., predicting commodity futures with stolen trading algorithms) or to understand the limitations of defensive models (EDRs, SIEMs) for planning evasive actions.
• Lateral Movement & Privilege Escalation: The ultimate goal for red teamers is to leverage MLOps platform access for deeper network penetration, moving laterally and escalating privileges within the target organization's wider infrastructure.

MLLo Kit: The Offensive Toolkit

The MLLo Kit is designed to abstract the complexity of interacting with MLOps platform APIs. It leverages stolen credentials like API keys or access tokens to perform actions. In its initial release, the tool supports three commercial platforms: Azure ML, BigML, and Google Cloud Vertex AI, offering six different attack modules, with plans for future expansion.

Platform-Specific Technical Details:

1. Azure ML:

• Architecture: Centered around a workspace, which provides access to datasets, models, and ML pipelines.
• Authentication: Multiple options, including web portal (Azure AD), service principal, Azure CLI, managed identities, and access tokens for REST API interaction. The speakers note that there's extensive research on stealing Azure access tokens (e.g., via social engineering, user workstations).
• Logging: Not enabled by default. Diagnostic logging must be explicitly enabled, with audit logs selected to be sent to a Log Analytics Workspace for detection rule development.
• REST API Abuse: The white paper details raw REST API calls for reconnaissance (listing subscriptions, workspaces, models) and model/data extraction. MLLo Kit streamlines these, allowing for validation of tokens (check module), listing projects (list_projects), and listing/downloading models (list_models, download_model) by their model ID. Model files are typically stored in Azure Storage blob accounts.

1. BigML:

• Architecture: Structured around organizations (one-to-many), each containing projects (one-to-many). Projects hold data sources (raw data), which are transformed into datasets (structured data) for model training (e.g., classification, regression models).
• Authentication: Web interface (MFA supported), API key for REST API, and a Python-based command-line tool (BigMLer). MLLo Kit abuses the API key authentication.
• Logging: Requires a private deployment to enable logging capabilities, which means cloud-based BigML instances typically lack logging for these types of attacks. This was a limitation for the research in terms of developing detection guidance.
• REST API Abuse: The white paper provides detailed curl commands. MLLo Kit uses the API key to perform reconnaissance and data/model extraction.

1. Google Cloud Vertex AI:

• Architecture: Organized by projects, each potentially containing an ML pipeline that consumes datasets and other input files to produce models.
• Authentication: Web console, Google Cloud CLI, and application default credentials. Similar to Azure, credentials can be obtained via social engineering, internal file shares, or wikis.
• Logging: Not enabled by default. Requires enabling Vertex AI data access audit logs and Google Cloud Storage logs, as GCS is the backing storage for models and datasets.
• REST API Abuse: MLLo Kit simplifies complex API interactions. The process for model extraction is particularly involved: first, the model must be exported from the model registry to a Google Cloud Storage bucket, and then downloaded from that GCS bucket to the attacker's machine. MLLo Kit handles this multi-step process programmatically, maintaining the directory structure of the downloaded files.

In essence, the technical deep dive reveals that MLOps platforms, despite their advanced capabilities, often exhibit fundamental security weaknesses common to many new technologies: insecure defaults, complex logging configurations, and an underdeveloped defensive ecosystem, all of which MLLo Kit is designed to exploit and highlight.

Demo / Proof of Concept

▶ Watch: Primary goals: open-source tool and inspiring defense (7:00)

The talk featured live demonstrations using MLLo Kit to illustrate practical attack scenarios against Azure ML and Google Cloud Vertex AI. While a demo for BigML was prepared, it was skipped during the live presentation due to time constraints.

Azure ML Model Extraction Demo

Scenario: An attacker successfully executes an Azure device code phishing attack against a data scientist, obtaining a valid Azure access token. This token grants access to Azure ML resources.

Attack Steps (using MLLo Kit):

1. Token Validation: The attacker first uses mllo_kit check --platform azure --credential <access_token> to validate the stolen access token. The tool confirms access and lists the Azure subscriptions the token is associated with. The subscription ID is a crucial piece of information for subsequent commands.
2. Workspace Reconnaissance: Next, the attacker uses mllo_kit list_projects --platform azure --subscription-id <subscription_id> to enumerate Azure ML workspaces (also referred to as projects) accessible within that subscription. For each workspace, MLLo Kit provides details including the resource group and region.
3. Model Listing: With the workspace details (name, resource group, region), the attacker lists available models in the model registry using mllo_kit list_models --platform azure --workspace-name <workspace_name> --resource-group <resource_group> --region <region>. This command reveals registered models, their types (e.g., MLflow models), and critically, their model IDs.
4. Model Exfiltration: The attacker selects a target model ID and executes mllo_kit download_model --platform azure --model-id <model_id> --workspace-name <workspace_name> --resource-group <resource_group> --region <region>. MLLo Kit then interacts with the Azure Storage blob accounts (the backing store for the model registry) to download the serialized model file (e.g., a .pickle file) and any associated dependencies or environment files (e.g., Python requirements.txt or Conda environment files). The tool automatically creates a directory and stores all files, maintaining the original structure.

Detection: The speakers noted that model download activity would be logged in the AML Models event schema with specific operation lists, which are detailed in the white paper for building KQL queries.

BigML Demo (Acknowledged but Skipped)

Scenario: An attacker discovers an API key for BigML on a publicly exposed source code repository.

Potential Attack: Using MLLo Kit, the attacker could leverage this API key to access the BigML instance, list organizations, projects, data sources, datasets, and models, and then proceed to steal private training data sets or models, which might contain sensitive information.

Google Cloud Vertex AI Model Extraction Demo

Scenario: An attacker performs a phishing attack, gains a command-and-control (C2) beacon, escalates privileges (e.g., via Active Directory), and laterally moves to an ML engineer's workstation. From this workstation, the attacker steals a Google Cloud access token (often found in an SQLite database used by the Google Cloud CLI).

Attack Steps (using MLLo Kit):

1. Token Validation & Project Listing: The attacker uses the stolen access token with mllo_kit list_projects --platform gcp --credential <access_token> to validate the token and enumerate accessible Vertex AI projects. The project ID is copied for subsequent actions.
2. Model Listing: The attacker then lists models within the target project's model registry using mllo_kit list_models --platform gcp --project-id <project_id>. This reveals various models, often associated with "experiment runs," and their respective model IDs.
3. Model Exfiltration (Complex Process): This is the most intricate part of the attack. Vertex AI does not allow direct download of models from the registry. Instead, the model must first be exported to a Google Cloud Storage (GCS) bucket. MLLo Kit handles this by:

• Identifying accessible GCS buckets.
• Initiating the export of the chosen model to a GCS location.
• Once exported, MLLo Kit then downloads all associated model files from the GCS bucket to the attacker's local machine, potentially proxied through the C2 implant to avoid direct download to the victim's machine. The tool ensures the directory structure is preserved, yielding the serialized model file and its dependencies.

Detection: The white paper outlines specific methods to filter for this attack activity, particularly focusing on Vertex AI data access audit logs for model export operations and Google Cloud Storage logs to detect when serialized model formats are touched or downloaded from GCS buckets.

These demos vividly illustrate how MLLo Kit provides a practical, automated way for red teams to simulate sophisticated attacks against real-world MLOps platforms, thereby exposing critical security vulnerabilities that organizations must address.

Defensive Implications

▶ Watch: Novelty: first public tooling and API abuse talks (8:10)

The research underscores a critical need for organizations to proactively secure their MLOps environments. Given the significant attack surface and the high value of compromised data and models, a robust defensive strategy is imperative.

General Security Best Practices:

• ML Asset Inventory: Maintain a comprehensive inventory of all ML assets, including models, datasets, and pipelines.
• Threat Modeling: Incorporate MLOps scenarios into existing threat modeling exercises. Consider both model-centric attacks and, more importantly, attacks against the underlying MLOps infrastructure and integrated systems. Assume that if sensitive data or actions (e.g., wire transfers) are involved, attackers will target the backend infrastructure.
• Data and ML Workflow Security Best Practices: Adhere to established security best practices for data handling, storage, and processing within ML workflows.
• C2 Beaconing Detection: Be vigilant for Command and Control (C2) beaconing originating from training and tuning environments, as this can indicate compromised developer workstations or MLOps infrastructure.
• Lateral Movement and Privilege Escalation Detection: Implement detections for lateral movement and privilege escalation activities within and between MLOps platforms and the wider enterprise network.

Platform-Specific Configuration Guidance:

• Azure ML:
• Enable Audit Logging: Crucially, enable diagnostic logging within Azure ML and ensure that audit logs are checked and shipped to a Log Analytics Workspace for analysis and custom rule development.

• BigML:
• Enable MFA: Multi-Factor Authentication (MFA) is often not enabled by default; ensure it is enforced for all users.
• Credential Rotation: Implement frequent credential rotation for API keys and other access tokens.
• Granular Access Controls: Apply the principle of least privilege. Avoid granting broad administrative access (e.g., "admin to everyone") to developers or users within BigML.

• Google Cloud Vertex AI:
• IAM for Roles and Groups: Leverage Google Cloud's Identity and Access Management (IAM) to manage roles and groups, implementing highly granular access controls.
• Security Command Center: Implement Google Cloud Security Command Center and configure it to monitor Vertex AI resources.
• Enable Audit Logs: Enable Vertex AI data access audit logs and Google Cloud Storage logs, as GCS is often the backing store for ML assets.

Detection Guidance (Leveraging MLLo Kit for Testing):

The white paper accompanying MLLo Kit provides detailed threat hunting queries and KQL (Kusto Query Language) queries for Azure ML, as well as recommended methods for Google Cloud Vertex AI.

• Azure ML: Monitor the AML Models event schema and associated operations for suspicious model access and download activities.
• BigML: Due to the requirement for a private deployment for logging, specific detection guidance for BigML's cloud offering was not developed.
• Google Cloud Vertex AI: Filter for specific method names related to model export and download activities. Additionally, monitor Google Cloud Storage logs for touch events on serialized model formats.

MLLo Kit for Defense:

MLLo Kit is not just an offensive tool; it's a defensive enabler. Organizations can use it to:

• Simulate Attacks: Test the efficacy of existing security controls and identify gaps.
• Develop Detections: Build and validate custom detection rules (e.g., YARA rules, Snort rules for hardcoded user strings in MLLo Kit, Sentinel rules, Google Cloud alert policies) against the types of activities MLLo Kit performs.
• Train IR Teams: Familiarize IR and threat hunting teams with the unique telemetry and attack patterns associated with MLOps compromises.

Ultimately, securing MLOps requires a shift in mindset. Defenders must understand that the rush to adopt AI creates a new, juicy target for attackers. By embracing tools like MLLo Kit and prioritizing comprehensive security practices, organizations can move from a reactive to a proactive stance, ensuring the integrity and confidentiality of their AI initiatives and the sensitive data they process.

Key Takeaways

• MLOps Security is Critically Overlooked: The security of the broader MLOps ecosystem, encompassing training environments, platforms, APIs, and integrations, is often neglected despite its direct access to critical enterprise data.
• Offensive Tooling and Research are Essential: The lack of public tools like MLLo Kit previously hindered effective red teaming and defensive development in MLOps. Such tools are vital for simulating real-world attacks.
• Default Insecurity is Pervasive: Crucial security features, especially diagnostic and audit logging, are frequently disabled by default across major commercial MLOps platforms, leaving organizations blind to attacks.
• Threat Actors Have Strong Motivations: From resource exhaustion and supply chain attacks (backdoored models) to financial fraud and nation-state espionage, adversaries have significant incentives to target MLOps platforms for data, model, and intellectual property theft.
• Comprehensive Threat Modeling is Required: Organizations must expand their threat modeling beyond model-centric vulnerabilities to include the entire MLOps pipeline, focusing on lateral movement, privilege escalation, and data exfiltration from the underlying infrastructure.
• Proactive Defense is Paramount: Defenders must enable and ship audit logs, implement granular access controls (least privilege), perform regular credential rotation, and actively develop and test detection rules against MLOps-specific attack patterns.

About the Speaker(s)

Chris Thompson leads the IBM X-Force Red team. He founded the adversary services (red team) there approximately nine years ago. Chris is an experienced speaker at numerous security conferences, serves on the Black Hat review board, and is the co-founder of an upcoming offensive AI conference.

Brett Hawkins is a team lead at IBM X-Force Red. He has developed a number of open-source tools and presented at various conferences. His public security research, including white papers, presentations, tools, and blog posts, is cataloged on his personal URL.