Jupysec: Auditing Jupyter to Improve AI Security

Joe Lucas (Offensive Security Researcher for AI · NVIDIA)

TROOPERS23 · Day 1 · Defense & Management · Defense & Management

Overview

In this insightful TROOPERS talk, Joe Lucas, an Offensive Security Researcher for AI at NVIDIA and a member of the Jupyter Security Council, demystifies the practicalities of securing AI development today. Moving beyond speculative future threats, Lucas focuses on the immediate and tangible security challenges within Machine Learning Operations (MLOps), specifically highlighting the often-overlooked vulnerabilities inherent in Jupyter environments. The presentation introduces Jupysec, a novel auditing tool designed to help organizations assess and improve the security posture of their Jupyter installations.

Watch on YouTube · Slides

Visual summary for Jupysec: Auditing Jupyter to Improve AI Security by Joe Lucas
Visual summary for Jupysec: Auditing Jupyter to Improve AI Security by Joe Lucas

Key moments

  1. 0:00 Introduction: Practical AI Security and Jupyter's Role
  2. 9:45 Jupyter Architecture Explained and its Security Implications
  3. 16:00 Demo: Runtime Model Poisoning without Disk Changes
  4. 18:30 Demo: Code Execution and Persistence via Notebook Startup
  5. 21:00 Jupysec: Automating Jupyter Security Audits
  6. 23:30 Jupysec Demo: Security Report and Configuration Findings
  7. 29:30 Q&A: Mitigating Jupyter Risks with Containers and Hub

Jupysec: Auditing Jupyter to Improve AI Security

Speakers: Joe Lucas, Offensive Security Researcher for AI, NVIDIA

Conference: TROOPERS

YouTube: https://www.youtube.com/watch?v=7HpuPmNLYjk

Overview

In this insightful TROOPERS talk, Joe Lucas, an Offensive Security Researcher for AI at NVIDIA and a member of the Jupyter Security Council, demystifies the practicalities of securing AI development today. Moving beyond speculative future threats, Lucas focuses on the immediate and tangible security challenges within Machine Learning Operations (MLOps), specifically highlighting the often-overlooked vulnerabilities inherent in Jupyter environments. The presentation introduces Jupysec, a novel auditing tool designed to help organizations assess and improve the security posture of their Jupyter installations.

Lucas argues that while algorithmic AI attacks garner significant attention, many critical security issues stem from fundamental application security principles and common misconfigurations within the MLOps toolchain. Jupyter, a ubiquitous platform for data scientists and AI developers, plays a central role across the entire ML lifecycle, from ideation and data processing to model training and even serving. This pervasive use, coupled with a complex and often misunderstood configuration landscape, makes Jupyter a prime target for attackers seeking to compromise AI systems. The talk underscores the necessity for security professionals to engage with conventional software development communities and "shift left" security efforts into the early stages of AI development.

Background

▶ Watch: Introduction: Practical AI Security and Jupyter's Role (0:00)

The rapidly evolving field of AI security encompasses a broad spectrum of risks, which Lucas categorizes into compliance issues (e.g., GDPR violations from compiled data), technical risks (traditional application security vulnerabilities), and reputational risks (e.g., AI jailbreaks leading to undesirable outputs). While model vulnerabilities like adversarial examples or data extraction techniques receive significant research focus, Lucas emphasizes that fundamental application security (AppSec) principles are often easier and more effective for achieving operational security objectives. He notes that compromising an AI system through an unauthenticated API to swap a model is frequently more straightforward than intricate algorithmic manipulations.

Jupyter, an open-source software ecosystem, originated for Julia, Python, and R, but now supports numerous kernels for languages like C++, SQL, and Lua. It started with single-panel notebooks but has evolved into more comprehensive environments like JupyterLab and JupyterHub. The core architecture involves a user interface in the browser communicating via HTTP and WebSockets with a Jupyter server, which then dispatches commands through a message queue to a kernel—the user's runtime environment. This design decouples the UI from the compute resource, allowing data scientists to run code on powerful remote machines from basic local devices. However, this flexibility introduces significant security challenges. Crucially, Jupyter Notebooks and Lab are designed as single-user applications with no inherent trust boundary between users. This is a critical point, as Lucas frequently observes organizations misconfiguring and sharing these instances among multiple users, creating a significant security exposure. Despite the existence of JupyterHub for multi-user deployments, misconfigurations remain a prevalent issue, with Verizon's 2022 data breach report attributing 21% of breaches to misconfiguration. The expansive and often nested configuration space of Jupyter, involving multiple file types (JSON, Python), command-line flags, and environment variables, further exacerbates this problem, making it a "nightmare" for administrators to secure effectively.

Key Findings

▶ Watch: Demo: Runtime Model Poisoning without Disk Changes (16:00)

The core findings of the talk revolve around demonstrating how Jupyter's inherent design and complex configuration space create significant vulnerabilities that can be exploited to compromise AI development workflows. Lucas highlights that traditional AppSec vulnerabilities and misconfigurations are often more impactful and easier to exploit than advanced algorithmic attacks in the context of MLOps. He identifies three primary attack primitives achievable within a compromised Jupyter environment, demonstrating how they can lead to data exfiltration, model poisoning, and persistent code execution:

  1. Silent Runtime Injection: Attackers can manipulate variables within a user's active Jupyter session without any visual indication in the UI. This allows for redirection of sensitive operations, such as changing an S3 bucket destination for data export, leading to exfiltration.
  2. In-Memory Data Poisoning: By injecting code into the user's runtime, an attacker can modify how data is processed or labeled in memory before it reaches the model training pipeline. This enables sophisticated model poisoning attacks without altering any files on disk, making detection challenging through traditional file integrity checks.
  3. Arbitrary Code Execution and Persistence on Startup: Jupyter environments can be configured to execute arbitrary code automatically when a notebook is opened or an instance starts. Leveraging directories like .ipython, attackers can establish persistent footholds, enabling command and control (C2) callbacks or other malicious activities that impact a data scientist's entire workflow.

To counter these threats, Lucas introduces Jupysec, an open-source auditing tool. Jupysec addresses the challenge of assessing the security posture of a running Jupyter instance, especially given the complexity of its configuration. It functions as both a Jupyter extension and a standalone Python library, designed to:

  • Identify deviations from secure default configurations.
  • Scan command history for indicators of compromise (IOCs).
  • Inspect important values in the running configuration (in memory) to uncover real-time vulnerabilities or attacks.

Jupysec aims to provide administrators and data scientists with actionable insights into their Jupyter environment's security, bridging the gap between theoretical vulnerabilities and practical defensive measures.

Technical Deep Dive

▶ Watch: Demo: Code Execution and Persistence via Notebook Startup (18:30)

Jupyter's architecture, while offering powerful flexibility, also presents a substantial attack surface. The system fundamentally separates the user's browser-based UI from the computational backend, which runs a Jupyter server that communicates with various kernels. Each kernel represents a runtime environment for a specific language (Python, R, Julia, etc.). This decoupling means that a data scientist can be running computationally intensive tasks on a remote supercomputer while interacting with it via a lightweight browser interface. This is enabled by sending commands over HTTP and WebSockets to the server, which then dispatches them to the appropriate kernel via a message queue.

The complexity of securing Jupyter stems largely from its vast and intricate configuration space. As Lucas illustrates, configuration data can reside in numerous files (some JSON, some Python), be overridden by command-line arguments during startup, or influenced by environment variables. These configurations are often nested, with Python configuration files calling others, creating a "1500-line nightmare" for administrators to parse and secure, as shown with a default Jupyter Hub configuration. This complexity makes it difficult for even experienced security professionals to ascertain the true running configuration of a Jupyter instance.

Lucas demonstrates several potent attack vectors:

  1. Runtime Variable Injection (0:14:00): This attack leverages the interactive nature of Jupyter. An attacker, with appropriate privileges (e.g., access to the same shared single-user JupyterLab instance), can inject code that redefines a critical variable. For instance, if a data scientist defines S3_BUCKET = "my-secure-bucket", an attacker can execute S3_BUCKET = "attacker-controlled-bucket" in the same kernel. Subsequent code executed by the unsuspecting data scientist, such as upload_data(S3_BUCKET, data), would then exfiltrate data to the attacker's bucket. Crucially, this happens without any visual change in the user interface, making it extremely stealthy.
  1. In-Memory Data Poisoning (0:16:00): This demonstration showcases a critical primitive for training-time attacks. Using the PyTorch library, Lucas illustrates how an attacker can manipulate the Dataset abstraction. In the example, the MNIST dataset (handwritten digits) is loaded. Normally, iterating through this dataset would yield an image and its correct label (e.g., a "5" image labeled as 5). By injecting code into the kernel, the attacker can redefine the dataset's __getitem__ method to always return a specific, incorrect label (e.g., 3), regardless of the actual image. If a machine learning engineer then trains a model using this compromised dataset, the model will learn to classify everything as 3. This attack operates entirely in memory, leaving no trace on disk files, thus bypassing traditional forensic methods focused on file integrity.
  1. Arbitrary Code Execution on Startup (0:19:00): This vulnerability exploits intended functionality for developer convenience. Jupyter environments often automatically execute scripts or commands from specific directories, such as the user's .ipython profile directory, upon startup. Lucas demonstrates this by starting a Docker container running JupyterLab, then setting up a Sliver C2 HTTP listener. By dropping a malicious script into the .ipython startup directory, he shows that simply opening a new notebook in the compromised Jupyter instance triggers a callback to the Sliver C2 server, granting the attacker a persistent shell on the host. This provides a powerful mechanism for initial access, persistence, and impact on a data scientist's workflow.

Jupysec is presented as a direct response to these challenges. It's designed to be lightweight and portable, allowing it to be easily deployed for reconnaissance or auditing. As a JupyterLab extension, it adds a "Security" row to the launcher and provides a "Security Report." This report, updated for JupyterLab 4.0, offers an at-a-glance view of critical configuration parameters, such as the allow_all_authorizer (a clear indicator of weak authorization), cross-site request forgery (CSRF) checks, remote access settings, and the status of collaborative mode (which Lucas notes "terrifies" him due to its potential for abuse). Jupysec also generates "findings" based on known vulnerabilities, suspected malicious activities (e.g., code in startup directories), and changed configuration values. The tool can also be used as a standalone Python library, allowing for auditing without enabling potentially risky extensions. Lucas acknowledges the inherent conflict of writing a security extension when general advice is to disable extensions, hence the standalone library option.

Demo / Proof of Concept

▶ Watch: Jupysec Demo: Security Report and Configuration Findings (23:30)

The talk features several compelling live demonstrations that visually underscore the vulnerabilities discussed:

  1. Silent Runtime Injection (0:14:00): Lucas first demonstrates a simple Python variable assignment within a Jupyter notebook (my_variable = "expected_value"). He then shows how an attacker, by executing code in the same environment, can silently redefine my_variable to "malicious_value" without any visible change to the data scientist's active notebook cells. This illustrates how an attacker could redirect sensitive operations, such as writing to an S3 bucket, to an unauthorized location.
  1. In-Memory Data Poisoning (0:16:00): This demo is particularly impactful for AI security. Lucas imports PyTorch and loads the MNIST dataset. He shows how iterating through the dataset initially yields correct image-label pairs (e.g., an image of '5' labeled as 5). He then injects a small piece of Python code that modifies the dataset's internal logic. After this injection, iterating through the same dataset now consistently returns an incorrect label (e.g., all images are labeled as 3), even though the underlying disk files remain untouched. This demonstrates a powerful primitive for model poisoning, where an attacker can subtly influence the training data to manipulate the model's behavior.
  1. Arbitrary Code Execution and C2 Callback (0:19:00): Lucas sets up a Docker container running JupyterLab and a Sliver C2 HTTP listener. He then places a simple Python script designed to establish a C2 connection into a Jupyter startup directory (e.g., .ipython/profile_default/startup). When he opens a new notebook in the JupyterLab instance, the script automatically executes, and a callback is immediately received on the Sliver C2 listener. This provides a clear proof-of-concept for achieving persistent code execution and remote control over the Jupyter host.
  1. Jupysec Auditing Tool (0:23:00): The final demonstration showcases the Jupysec JupyterLab extension. Lucas navigates to the JupyterLab launcher, where a new "Security" row is visible. Clicking on "Security Report" opens a panel displaying key configuration details at a glance, such as whether an allow_all_authorizer is enabled, the status of CSRF checks, and remote access settings. The report also lists "findings," categorizing them into potential vulnerabilities, indicators of compromise (IOCs), and changed configuration values. For example, it might highlight the presence of code in a startup directory or a modified default setting, providing actionable security intelligence directly within the Jupyter environment.

These demonstrations effectively transition from theoretical vulnerabilities to concrete, exploitable scenarios, making a strong case for the practical security challenges within Jupyter environments.

Defensive Implications

▶ Watch: Q&A: Mitigating Jupyter Risks with Containers and Hub (29:30)

The insights from this talk provide critical guidance for defenders operating in MLOps environments. The primary takeaway is that securing AI development often begins with robust application security practices and diligent configuration management for ubiquitous tools like Jupyter.

  1. Prioritize Configuration Management: Given Jupyter's vast and nested configuration space, organizations must establish stringent processes for managing Jupyter configurations. This includes using version control for configuration files, implementing peer review for changes, and strictly controlling who can modify configuration options, command-line flags, and environment variables. Tools like Jupysec can help audit these configurations against best practices and detect deviations.
  1. Enforce User Isolation: Jupyter Notebooks and Lab are explicitly designed as single-user applications. For multi-user environments, organizations must deploy JupyterHub or similar multi-tenant solutions that provide proper user isolation and authentication. Sharing single-user JupyterLab instances is a severe security risk, enabling runtime injection and other attacks between users. Containerization and virtualization (e.g., Docker, Kubernetes) are excellent strategies to further isolate individual Jupyter environments and limit the blast radius of a compromise, as suggested by an audience question.
  1. Audit Running Instances: Relying solely on static configuration file analysis is insufficient. Defenders need to regularly audit the running configuration of Jupyter instances, as command-line arguments and environment variables can override file-based settings. Tools like Jupysec are invaluable here, providing a way to inspect in-memory settings, command history, and detect indicators of compromise. This proactive auditing helps identify misconfigurations and active compromises that might otherwise go unnoticed.
  1. Enhance Logging and Monitoring: The talk highlights that in-memory attacks like data poisoning leave no disk-based traces. This necessitates enhanced logging and monitoring capabilities. While Jupyter has some application-level logging, organizations should integrate Jupyter logs with broader security information and event management (SIEM) systems. Lucas mentions a community movement to integrate with Zeek for better exported logging. Defenders should also monitor for unusual process execution, network connections (especially C2 callbacks), and changes to critical Jupyter directories or configuration files.
  1. Be Cautious with Extensions and Startup Scripts: While extensions and startup scripts enhance developer experience, they also introduce significant attack surfaces. Defenders should implement strict policies regarding the use and installation of Jupyter extensions, vetting them for security vulnerabilities. Similarly, while .ipython startup directories are intended for automation, they must be tightly controlled and monitored for unauthorized modifications. Organizations should differentiate between essential, vetted automation and potentially malicious code.
  1. Shift Left Security and Developer Education: Security should be integrated early in the MLOps lifecycle ("shift left"). Data scientists, often focused on functionality, may inadvertently introduce security risks. Providing them with knowledge and user-friendly tools like Jupysec, along with secure-by-default environments and easy-to-access JupyterHub instances, encourages secure practices without stifling innovation. This also involves educating developers on the security implications of shared environments, configuration options, and the risks associated with certain functionalities.
  1. Consider WebAssembly Kernels: For highly sensitive applications or untrusted code execution, Lucas suggests that WebAssembly-based kernels offer a promising future for sandboxed execution directly in the browser, providing a strong isolation boundary.

By adopting these defensive strategies, organizations can significantly harden their Jupyter environments, mitigating critical risks that could otherwise undermine the integrity and confidentiality of their AI development efforts.

Key Takeaways

  • Jupyter's Critical Role and Vulnerability: Jupyter is ubiquitous across the MLOps lifecycle, making its security paramount. Its flexible architecture, while powerful, introduces significant attack surface, often overlooked in favor of more complex AI-specific threats.
  • Misconfigurations as Primary Attack Vectors: Practical AI security often hinges on addressing fundamental application security issues and misconfigurations, which are frequently easier to exploit and more impactful than advanced algorithmic attacks. Verizon reports 21% of breaches from misconfiguration.
  • Stealthy Attack Primitives: Attackers can achieve silent runtime variable injection, in-memory data poisoning (e.g., changing MNIST labels without disk access), and persistent code execution via startup scripts (e.g., Sliver C2 callback), all of which can severely compromise AI development.
  • Jupysec for Auditing and Detection: Jupysec provides a crucial auditing capability, available as a Jupyter extension and Python library, to assess the security posture of running Jupyter instances, detect deviations from secure defaults, and identify indicators of compromise.
  • Importance of Isolation and Configuration Management: Secure Jupyter deployments require strict user isolation (using JupyterHub for multiple users, containerization for environments) and rigorous management of its complex, nested configuration space to prevent widespread compromise.
  • Shift Left Security and Developer Empowerment: Integrating security early in the MLOps process and empowering data scientists with knowledge and tools like Jupysec can significantly improve the overall security posture, balancing innovation with necessary controls.

About the Speaker(s)

Joe Lucas is an Offensive Security Researcher for AI at NVIDIA, where he is part of the NVIDIA AI Red Team. His team is tasked with enhancing the security of NVIDIA's AI products, services, and development through adversarial testing. Lucas is also a member of the Jupyter Security Council, an open-invitation body dedicated to improving the security of the broader Jupyter open-source ecosystem. He actively advocates for security professionals to engage with conventional software development communities, emphasizing the importance of "moving the needle on security" by presenting at developer-focused conferences like JupyterCon. His work focuses on practical, actionable security measures for AI systems currently in development.

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

Joe Lucas actually gets it. While the rest of the industry is jerking off over prompt injections and theoretical adversarial math, he's exploiting the real attack surface: the loosely configured Jupyter environments your data scientists leave wide open. This is a highly practical MLOps security talk with a tool drop that matters.

Heather Calloway (CISO) — SOLID

This talk cuts through the noise of AI security hype to target the actual weak link in the chain: the data science development environment. Lucas demonstrates that attackers don't need complex algorithmic exploits to poison a model if the Jupyter instance building it is misconfigured. Worth your architecture and MLOps teams' time immediately.

→ Top-rated talks at TROOPERS23

All talks from TROOPERS23