Confusing CNNs on NPUs with LLMs using FI; or, How Much AI Can We Fit?

Confusing CNNs on NPUs with LLMs using FI; or, How Much AI Can We Fit?

Speakers: Jasper van Woudenberg (Senior Principal Security Technologist, Keysight); Rajesh Velegalati (Principal Security Analyst, Keysight)

Conference: Hardwear.io USA

YouTube: https://www.youtube.com/watch?v=AHzcj1Zbsjo

Overview

This groundbreaking talk, delivered by Jasper van Woudenberg and Rajesh Velegalati from Keysight, delves into the critical and emerging field of AI hardware security. The presentation showcases a two-pronged attack: first, demonstrating the vulnerability of Neural Processing Units (NPUs) running Convolutional Neural Networks (CNNs) to traditional fault injection techniques, leading to alarming misclassifications and "hallucinations." Second, and perhaps more significantly, it introduces an innovative framework where Large Language Models (LLMs) are leveraged as intelligent agents to automate and optimize complex fault injection campaigns, effectively pitting AI against AI.

The talk highlights a pivotal shift in cybersecurity, moving beyond software vulnerabilities to address the hardware underpinnings of AI systems. As AI models become ubiquitous and increasingly deployed on specialized hardware like NPUs, understanding and mitigating their physical attack surfaces becomes paramount. Van Woudenberg and Velegalati not only expose these vulnerabilities but also demonstrate how advanced AI tools, accessible to anyone with the right lab equipment, can accelerate the discovery of such flaws, posing a significant challenge for defenders.

The research presented is crucial because it illustrates the growing sophistication of attack methodologies. By showing that LLMs can not only drive complex hardware interactions but also perform advanced analysis and even binary patching to optimize attacks, the speakers underscore the urgent need for robust hardware security measures in the AI era. This work serves as a stark warning and a call to action for the security community to prepare for a future where hardware, rather than just software, becomes the "next weakest link" in the cybersecurity chain.

Background

▶ Watch: Demo: Drone object recognition confused by fault injection (1:00)

The genesis of this research stems from the rapid advancements in AI, particularly the emergence of highly capable models like GPT-4. Jasper van Woudenberg, having a long-standing background in AI before its mainstream surge, recognized an opportunity to apply these powerful new tools to the domain of security, specifically hardware security. Despite their enthusiasm for AI's potential, both speakers, as seasoned security professionals, maintained a healthy skepticism regarding its limits and potential failure modes. This led them to investigate how AI systems, particularly those accelerated by dedicated hardware, could be broken.

The core problem addressed is the security of Convolutional Neural Networks (CNNs) running on Neural Processing Units (NPUs). CNNs are a class of deep neural networks commonly used for tasks like object recognition, where they take pixel data as input and output classifications, bounding boxes, and confidence scores. NPUs are specialized processors designed to accelerate these computationally intensive AI workloads, making them faster and more efficient than general-purpose CPUs.

Traditionally, fault injection (FI) has been a powerful technique in hardware security to induce errors in a device's operation by manipulating environmental factors like voltage, clock, or electromagnetic fields. While CNNs are inherently designed to be robust against minor input noise (e.g., slight variations in pixels), their internal computations, particularly in the later stages where information becomes more discrete (e.g., object labels, confidence scores), can be highly sensitive to precise faults. The challenge lies in identifying where and when to inject these faults to achieve a desired, exploitable outcome rather than just a system crash. The speakers highlight that the existence of this vulnerability, combined with the increasing reliance on NPUs for critical AI functions, creates a significant security gap that needs urgent attention.

Key Findings

▶ Watch: Introducing LLM agents to automate fault injection (10:00)

The research presented by van Woudenberg and Velegalati yielded several critical findings regarding the security of AI hardware and the potential of AI-driven attack automation:

1. Vulnerability of CNNs on NPUs to Fault Injection: The talk definitively demonstrated that voltage fault injection can successfully corrupt the operation of CNNs running on Rockchip NPUs. This leads to significant misclassifications and the "hallucination" of objects that are not present in the input image. For instance, the system was observed detecting non-existent skateboards with high confidence (e.g., 0.7), or even seeing "hot dogs" or "handbags" across an image.
2. Impact on Object Tracking Systems: The induced faults not only confused the single-frame object detection but also propagated to subsequent object tracking algorithms running on the CPU. This illustrates a cascading failure, where a hardware fault at the NPU level can destabilize higher-level AI applications.
3. LLMs as Powerful Fault Injection Agents: A central contribution is the successful development and demonstration of an LLM-driven framework for automating fault injection campaigns. This framework, composed of Supervisor and Optimizer GPTs interacting with custom hardware abstraction layers (5Pi and MCP servers), proved capable of autonomously navigating the complex parameter space of fault injection.
4. Significant Efficiency Gains: The LLM agent dramatically reduced the time and effort required for fault injection. While manual identification of sensitive parameters could take 22,000 attempts, the LLM-driven system consistently found exploitable faults within an average of 1,400-1,500 attempts (with a minimum of 700 and maximum of 2,400 over stress tests), representing a substantial speed-up and efficiency improvement.
5. Advanced LLM Capabilities in Attack Automation: Beyond parameter optimization, the LLM demonstrated surprising capabilities, including:

• System Health Monitoring: The Supervisor GPT could perform live checks like verifying LED status via camera or analyzing trigger signals from oscilloscopes.
• Side-Channel Analysis: The LLM, when prompted, successfully performed Welch's t-testing (similar to TVLA) on electromagnetic (EM) traces to precisely pinpoint critical code execution points, narrowing glitch windows from 5 microseconds to 20 nanoseconds.
• Binary Analysis and Patching: In a particularly striking example, the Codex model 5.5 analyzed a target binary, identified an issue with trigger signal length, disassembled the binary, patched it to expand the trigger, and re-tested it—all within a few prompts and achieving a five-fold speedup in the overall process.

1. Cost-Effectiveness of LLM-driven Attacks: The average cost for a successful fault injection outcome using the LLM agent was estimated at $3.95, consuming approximately 2.26 million prompt tokens, indicating a potentially cost-effective method for automating hardware security research or attacks.

Technical Deep Dive

▶ Watch: LLM optimizes trigger by disassembling and patching binary (22:00)

The technical exposition of the talk begins with a detailed look at the target hardware and the initial manual fault injection setup. The primary target was a Rockchip board equipped with an NPU, chosen for its common use in embedded AI applications. The CNN running on this NPU performed standard object recognition, processing camera frames at high rates (e.g., 30 frames per second) to output bounding boxes, labels, and confidence scores.

The fault injection method employed was voltage fault injection, specifically a "quick crowbar" technique. This involves momentarily dropping the power supply voltage to induce transient errors within the NPU's operation. To precisely time these glitches, the researchers used an EM probe to monitor the NPU's electromagnetic emissions, identifying specific phases of the CNN execution. Their strategy was to target the "backbone and head" – the latter, more discrete information processing stages of the CNN – as these were hypothesized to be less robust to faults than the initial pixel-processing layers.

Initial experiments involved a standard fault injection workflow: setting bounds, random glitching, and optimization. Visualizing this on a glitch length vs. glitch voltage plot, they identified a "rainbow shape" – a sensitive boundary between normal execution (green), system crashes (yellow), and the "interesting area" where faults manifested as unexpected behavior. Across 4,700 experiments, they observed various outcomes:

• Confidence shifts: Altering the confidence score of detected objects, either lowering (false sense of security) or increasing (false alarms).
• Hallucinations: The most compelling outcome, where non-existent objects were detected with varying confidence. Examples included a skateboard detected with 0.7 confidence, or widespread detection of "hot dogs," "handbags," or even "trucks" in blank images. These hallucinations were not uniformly distributed, suggesting potential biases in how the CNN could be manipulated.

The second part of the technical deep dive focuses on the LLM agent architecture designed to automate these complex fault injection campaigns. Raj Velegalati detailed a modular system:

1. Operator: A human analyst provides an overarching objective (e.g., "find a red outcome as fast as possible").
2. GPTs: Two distinct GPT models operate in concert:

• Supervisor GPT: Responsible for the overall health of the system. It monitors for issues, triages problems (e.g., "board is off," "trigger signal messed up"), and suggests remedies. Crucially, it gates the optimization process, deciding when the system is stable enough for the Optimizer to act.
• Optimizer GPT: Focuses on generating the next set of fault injection parameters. It requests evidence from the MCP servers and, based on the summarized data, proposes new glitch length, voltage, and delay values. Optimization only proceeds when the Supervisor confirms system validity.

1. 5Pi (FiPy) Inspector Module: This Python-based layer acts as the execution layer and a critical safety mechanism. It interfaces directly with the target hardware and the fault injection hardware, sending parameters and receiving data. The speakers explicitly stated their reluctance to give GPTs direct, unfettered control over hardware, hence the 5Pi intermediary. It waits for a synchronization/trigger signal from the target before inducing a glitch.
2. MCP (Measurement, Control, and Processing) Servers: These custom-coded servers are the interface to all lab instruments. They can communicate with oscilloscopes (for EM traces, trigger signals), cameras (for visual health checks like LED status), and relays (for power cycling). Critically, MCPs summarize raw data into a structured format before sending it to the GPTs, preventing the LLMs from being overwhelmed by useless raw data. It also includes a side-channel analysis module.

This architecture allows for a sophisticated loop: the Operator sets the goal, the GPTs analyze and strategize, 5Pi executes, MCP collects and processes evidence, and the GPTs then refine their approach based on this feedback. The speaker emphasized that successful LLM integration required significant "vibe-ing" and prompt engineering, injecting crucial FI domain knowledge and context into the LLMs to guide their decision-making.

Demo / Proof of Concept

▶ Watch: LLM agent efficiently finds faults on the NPU target (24:00)

The demonstration phase of the talk showcased the LLM-driven fault injection system in action, first on a known-good target and then on the NPU.

1. Initial Proof of Concept on "Pinata":

The system was initially tested on an internal Arm core target named "Pinata," used for training purposes. This target ran a simple "if pin check" algorithm, where a correct input pin was validated. The objective for the LLM was to find a "red outcome" – meaning the system should incorrectly validate a wrong pin as correct.

A color-coding system was established to categorize outcomes:

• Green: Expected outcome (wrong pin detected as wrong).
• Yellow: Mute or no response (glitched too hard, system crashed).
• Orange: Any other unexpected response.
• Red: The desired outcome (wrong pin detected as right).

The live demo, condensed from 6 minutes to 1 minute, showed the LLM starting from predefined "guardrails" for parameters. The system displayed the LLM's internal "thinking" – its current strategy, why it chose a particular move, and the expected effect. Live plots of glitch length vs. glitch voltage and glitch delay illustrated the LLM scanning the parameter space. It was observed efficiently narrowing down regions, identifying areas where glitches had no effect (all green), and then adjusting its parameters until it found a "red" outcome, at which point the campaign automatically stopped.

2. Supervisor's Health Checks:

The demo also highlighted the Supervisor GPT's robust health-checking capabilities. Examples included:

• Taking a camera snapshot of the board to verify if LEDs were on, confirming the board's liveness and responsiveness.
• Analyzing picoscope traces to check for the presence of a trigger signal. If a trigger was missing, the Supervisor could diagnose potential wiring issues (e.g., "look at your wire from your target to the spider, maybe that is messed up").
• Detecting persistent timeouts and reporting the board as "off" after 13 consecutive failures, prompting a physical check.

3. Advanced EM Signal Processing:

In a more advanced scenario on "Pinata," the LLM was capable of performing EM signal processing. Initially, it might report too much noise for basic analysis. However, it could then perform a "good versus bad comparison" by analyzing EM traces from correct and incorrect pin checks. When even this was insufficient, the LLM could initiate a Welch's t-testing (similar to TVLA in side-channel analysis). By comparing 60 good and 60 bad pin checks, it identified a t-spike between 470 and 490 nanoseconds, accurately pinpointing the exact time window where the pin check logic executed. This allowed it to narrow down a large 5-microsecond glitch window to a precise 20-nanosecond window, significantly improving targeting efficiency. It was noted, however, that this capability was a "soft nudge" and the LLM sometimes chose to ignore it, a lesson learned for future prompt engineering.

4. LLM Optimization and NPU Glitching:

The talk further demonstrated the LLM's ability to optimize its search strategy. Initially, with basic prompts, the LLM scanned parameter space inefficiently. By providing more FI domain knowledge (e.g., "you need to be in the transition boundary between green and yellow"), the LLM learned to focus its search on the sensitive curves, leading to much faster discovery of fault regions.

A remarkable example of the LLM's advanced capabilities was its interaction with a Codex model 5.5. When tasked with speeding up the fault injection process, the Codex model:

• Analyzed existing Python scripts and the target binary.
• Hypothesized that the trigger signal length was too narrow due to inference success without a hardware trigger.
• Pulled the binary from the board, disassembled it.
• Located the trigger signal generation point, expanded its length.
• Patched the binary back onto the board and re-tested.

This entire process, driven by "a couple of prompts," resulted in a five-fold speedup in fault injection attempts.

Finally, the LLM was set to glitch the actual object inference pipeline on the NPU, with the objective of finding a "red outcome" (hallucination or misclassification) as fast as possible. Within 800 attempts, it successfully found a red outcome. A stress test of nine fresh campaigns (without prior data or nudges) showed an average of 1,471 attempts to find a red outcome, ranging from 700 to 2,400 attempts. This was compared to the 22,000 attempts required for the manual identification phase, clearly demonstrating the LLM's superior efficiency in exploring the fault parameter space.

Defensive Implications

▶ Watch: Conclusions: AI as a tool for hardware security defense (30:00)

The findings presented in this talk carry profound implications for cybersecurity defenders, especially as artificial intelligence permeates more critical systems and specialized hardware. The core message is clear: hardware is the next weakest link.

1. Hardware as a Critical Attack Surface for AI: The demonstration that NPUs, the workhorses of AI acceleration, are vulnerable to physical attacks like fault injection is a significant concern. Unlike software vulnerabilities that can often be patched remotely or through updates, hardware flaws are far more difficult to remediate. A fix typically requires a new chip tape-out, a costly and time-consuming process that can take years. This means that vulnerabilities discovered today could persist in deployed systems for extended periods, creating long-term exposure.
2. Sophistication of AI-driven Attacks: The ability of LLMs to not only automate basic fault injection but also to perform complex tasks like binary analysis, patching, and side-channel analysis elevates the threat landscape. Attackers equipped with similar AI tools could accelerate their research, discover vulnerabilities more rapidly, and develop highly targeted exploits with unprecedented efficiency. This shifts the arms race, requiring defenders to also leverage AI for defense or risk being outpaced.
3. Need for Hardware Security in AI Design: Developers of AI hardware and systems must integrate security considerations from the earliest design stages. This includes designing NPUs and their surrounding circuitry to be robust against fault injection techniques. Countermeasures such as redundant computations, error detection and correction codes, and physical tamper detection mechanisms become increasingly vital.
4. Monitoring and Anomaly Detection: Defenders need to implement sophisticated monitoring systems capable of detecting anomalous behavior not just in software execution but also in the underlying hardware. This might involve monitoring power consumption, electromagnetic emissions, or NPU output for subtle deviations that could indicate a successful fault injection attack, even if the system doesn't crash.
5. Supply Chain Security: The initial scenario in the talk, involving a "mod chip" inserted into the supply chain, underscores the importance of securing the entire hardware supply chain. Fault injection attacks often require physical access, and compromised hardware at any stage of manufacturing or distribution could facilitate such attacks.
6. Accessibility of Attack Tools: Van Woudenberg explicitly stated that the tools and techniques demonstrated are publicly accessible, leveraging "any publicly accessible model" and requiring only "the toys in your lab." This democratization of advanced attack capabilities means that the barrier to entry for performing sophisticated hardware attacks is potentially lowering, making it a broader threat.
7. Bridging the Gap Between Software and Hardware Security: The talk serves as a call to action for the cybersecurity community to expand its focus beyond traditional software vulnerabilities to encompass the unique challenges of hardware security in the context of AI. As software layers become more secure, hardware will naturally become the "next weakest link," demanding a paradigm shift in defensive strategies.

Key Takeaways

• NPUs running CNNs are vulnerable to fault injection, leading to exploitable outcomes like misclassification and "hallucinations" of non-existent objects, which can also confuse higher-level object tracking systems.
• Large Language Models (LLMs) can be highly effective agents for automating and optimizing complex hardware security tasks like fault injection campaigns, drastically reducing the manual effort and time required to find vulnerabilities.
• An LLM-driven framework, comprising Supervisor and Optimizer GPTs, interacting with hardware abstraction layers (5Pi) and instrument controllers (MCP servers), can autonomously manage system health, analyze evidence, and refine attack parameters.
• LLMs demonstrate advanced capabilities, including binary analysis, patching binaries to optimize attack setups, and performing side-channel analysis (e.g., Welch's t-testing) to precisely target fault injection.
• Hardware security for AI systems is a critical and emerging challenge, as vulnerabilities are difficult to patch post-deployment (requiring chip tape-outs), making them long-lasting threats that defenders must proactively address.
• Effective integration of LLMs in security research requires careful prompt engineering, injection of domain-specific context, and structured, summarized feedback from instrumentation rather than raw data.

About the Speaker(s)

Jasper van Woudenberg is a Senior Principal Security Technologist at Keysight. He has a long-standing passion for Artificial Intelligence, holding a degree in the field from before its recent boom, though he has primarily focused on security throughout his career. His recent work involves exploring the intersection of AI and security, particularly in hardware.

Rajesh Velegalati is a Principal Security Analyst at Keysight. He brings extensive practical experience to the field, with over 10 years dedicated to performing fault injection experiments and analysis. His deep expertise in hardware-level attacks was instrumental in developing and validating the LLM-driven fault injection framework presented in the talk.