Confusing CNNs on NPUs with LLMs using FI; or, how much AI can we fit?
Jasper van Woudenberg (Keysight Device Security Testing)
[un]prompted.au 2026 · Day 1 · Main Hall
Overview
In his compelling talk at [un]prompted.au, Jasper van Woudenberg from Keysight Device Security Testing explored the vulnerabilities of Neural Processing Units (NPUs) to physical fault injection attacks and introduced a novel approach to automate this complex research using Large Language Models (LLMs). The presentation, titled "Confusing CNNs on NPUs with LLMs using FI; or, how much AI can we fit?", delved into how carefully timed voltage glitches can manipulate the output of Convolutional Neural Networks (CNNs) running on edge AI devices, leading to critical misclassifications and security bypasses.

Key moments
- 0:00 Introduction and demo of NPU confusion
- 2:30 Physical lab setup for fault injection
- 5:30 Pinpointing NPU execution for precise fault injection
- 10:00 Examples of AI model confusion from faults
- 12:00 Automating fault injection using LLM agents
- 18:00 LLM autonomously diagnoses and fixes hardware issues
- 21:00 Critical insights on benchmarking LLMs for effectiveness
- 25:00 Q&A: Discussing NPU resilience and real-world impact
Confusing CNNs on NPUs with LLMs using FI; or, how much AI can we fit?
Speakers: Jasper van Woudenberg (Keysight Device Security Testing)
Conference: [un]prompted.au
YouTube: https://www.youtube.com/watch?v=lobPveaww94
Overview
In his compelling talk at [un]prompted.au, Jasper van Woudenberg from Keysight Device Security Testing explored the vulnerabilities of Neural Processing Units (NPUs) to physical fault injection attacks and introduced a novel approach to automate this complex research using Large Language Models (LLMs). The presentation, titled "Confusing CNNs on NPUs with LLMs using FI; or, how much AI can we fit?", delved into how carefully timed voltage glitches can manipulate the output of Convolutional Neural Networks (CNNs) running on edge AI devices, leading to critical misclassifications and security bypasses.
Van Woudenberg demonstrated that these attacks can cause object recognition systems to hallucinate non-existent objects, ignore present ones, or even change confidence scores to effectively make detections disappear. Beyond exposing these hardware vulnerabilities, a significant portion of the talk focused on how LLMs, acting as both supervisors and optimizers, can dramatically accelerate and refine the traditionally laborious process of fault injection research. This dual focus highlights the evolving landscape of security: AI systems are becoming critical attack surfaces, while simultaneously offering powerful tools for security researchers.
The implications of this research are far-reaching. As AI becomes ubiquitous in edge devices, from drones and autonomous vehicles to smart cameras and industrial control systems, understanding and mitigating these physical layer attacks is paramount. The ability to reliably confuse an NPU without crashing the device presents a significant threat, potentially enabling sophisticated supply chain attacks or real-world sabotage. Furthermore, the successful integration of LLMs into the fault injection workflow marks a paradigm shift in hardware security testing, promising more efficient discovery of vulnerabilities and the development of robust countermeasures.
Background
▶ Watch: Introduction and demo of NPU confusion (0:00)
The talk begins by illustrating a common scenario: an object recognition pipeline in an edge AI device, such as a drone. This pipeline typically involves a camera capturing frames, a CPU performing initial processing (resizing, normalization), and a dedicated Neural Processing Unit (NPU) handling the intensive object recognition tasks using a Convolutional Neural Network (CNN). The NPU outputs bounding boxes and confidence scores for detected objects, which are then often fed into a tracking algorithm to stitch detections across multiple frames. This sequential architecture, while efficient, introduces multiple points of potential failure or manipulation.
Traditional hardware security research, particularly fault injection (FI), involves physically manipulating a device to induce abnormal behavior. Historically, this has been applied to CPUs to bypass security mechanisms like bootloaders or cryptographic operations. However, targeting NPUs, which are specialized accelerators for AI workloads, presents a unique challenge and a relatively underexplored area. The problem arises because modern AI models, while robust to minor input perturbations, can be highly sensitive to internal state corruption, especially in later layers of the network. The manual process of fault injection is notoriously tedious, requiring extensive parameter searches across glitch voltage, duration, and timing, often yielding inconsistent results or device crashes before useful faults are found.
The motivation for this research stems from the increasing reliance on AI in safety-critical and security-sensitive applications. If an NPU can be reliably confused, it could lead to catastrophic failures in autonomous systems, surveillance, or industrial automation. The existing landscape of AI security often focuses on adversarial examples (software-based attacks), but physical attacks on the underlying hardware represent a different, often more potent, threat vector. By exploring NPU vulnerabilities to FI, van Woudenberg and his team aim to provide crucial insights into protecting these emerging AI-powered systems.
Key Findings
▶ Watch: Pinpointing NPU execution for precise fault injection (5:30)
The research yielded several critical findings regarding the susceptibility of NPUs to fault injection and the efficacy of LLMs in automating this process.
Firstly, NPU fault injection proved remarkably effective at confusing Convolutional Neural Networks (CNNs) without causing system crashes. After approximately 4,700 focused experiments, distinct categories of faults were observed. The most prevalent effect was on detection scores, with confidence shifts occurring in 77.1% of cases. This means attackers could either drop an object's confidence below a detection threshold, effectively making it "disappear," or, less frequently, artificially inflate scores, leading to false positives. Examples included a bus image causing a skateboard to appear, or the system detecting "hot dogs everywhere," or even "handbags" as frequent false positives. A black image, when glitched, could produce a "truck at night."
A surprising finding was the robust stability of the System-on-Chip (SOC) under fault injection. Unlike many CPU fault injection experiments where device crashes are common, the NPU-equipped SOC demonstrated a high tolerance to these faults, reliably producing altered AI outputs without rebooting or halting. This stability makes NPU fault injection a more practical and insidious attack vector, as the system continues to operate, but with corrupted AI perception. The initial demonstration of a drone tracking a person, then becoming "confused" and tracking numerous non-existent objects, vividly illustrated this. The subsequent tracking algorithm further exacerbated the problem, maintaining confusion for extended periods, highlighting the cascading impact of NPU faults on downstream AI components.
Secondly, the talk showcased the transformative potential of LLMs in automating and optimizing the complex, multi-dimensional parameter search inherent in fault injection. An architecture was developed where FiPy, Keysight's proprietary fault injection software, handles the high-speed inner loop of glitching, while an LLM (specifically, a GPT model) operates in the outer loop, strategizing and optimizing the search. This setup included a Supervisor GPT responsible for system health monitoring (e.g., observing LEDs via camera, power cycling the entire system using relays) and an Optimizer GPT that, once the system was deemed stable by the supervisor, refined fault injection parameters.
The LLM proved significantly more efficient than manual methods, particularly when guided with appropriate hints. While initial LLM scans could be inefficient, providing context like "focus on the green-yellow transition boundary" (where faults are most likely) drastically improved its targeting. A particularly impressive demonstration involved a Codex 5.5 model diagnosing a missing trigger issue in the fault injection setup. It hypothesized the problem, then used SSH access to the target device, reverse-engineered the binary to locate the GPIO toggling code, extended the delay, and re-uploaded the modified binary, resolving the issue autonomously. This showcased the LLM's capacity for complex problem-solving and system-level debugging, moving beyond mere parameter optimization.
Finally, the research underscored the critical importance of benchmarking LLM performance in security research. Experiments with different LLM models (e.g., Sonnet vs. Haiku) and varying levels of guidance (baseline, toolkit, playbook, or both) for a key extraction task revealed counterintuitive results. For a more intelligent model like Sonnet, providing toolkits and playbooks sometimes increased the cost or time to achieve a target confidence, suggesting an "added mental load." Conversely, a less intelligent model like Haiku significantly benefited from clear instructions, becoming much more effective. This highlights that the optimal LLM strategy is highly dependent on the specific model and task, necessitating rigorous benchmarking to avoid misallocating resources or drawing incorrect conclusions about LLM efficacy.
Technical Deep Dive
▶ Watch: Automating fault injection using LLM agents (12:00)
The core of van Woudenberg's research lies in its meticulous technical execution, both in the physical fault injection setup and the innovative integration of LLMs.
The physical setup centers around a Rockchip Android development board, chosen for its integrated NPU accelerator. To induce faults, a custom fault injection device (a "big black box") is used to temporarily glitch the supply voltage. This involves momentarily shorting the power supply down to zero, a delicate operation where the duration and depth of the glitch are critical. Too long, and the device reboots; too short, and no effect is observed. The goal is to find a "sweet spot" that causes bit flips or brownouts without crashing the system. An oscilloscope is used for real-time monitoring of voltage lines and a small electromagnetic (EM) probe is placed over the chip to observe activity.
A crucial aspect of targeted fault injection is precise timing. The research team developed a method to synchronize their glitches with the NPU's operation. They run custom code on the target chip that pulls a General Purpose Input/Output (GPIO) line high just before sending a frame to the NPU for processing. This GPIO signal acts as an external trigger for the oscilloscope, allowing researchers to observe the NPU's execution waveform. To identify the most impactful timing window within the NPU's computation, an ablation study (or differential analysis) was performed. By running the CNN with and without its "head and neck" components (the later, more abstract layers responsible for final classification), they could visually identify the corresponding activity on the oscilloscope trace. Targeting these later stages of the CNN, where data has been transformed into more abstract categories and bounding box coordinates, was hypothesized to yield more significant and localized effects than corrupting raw input image data.
Fault injection is fundamentally a multi-dimensional parameter search. The key parameters explored were:
- Timing: When to inject the fault relative to the NPU's execution.
- Glitch Length: The duration of the voltage drop.
- Glitch Depth: How low the voltage drops.
This search space is often visualized using heatmaps, where axes represent glitch length and voltage, and colors indicate the observed outcome (green for normal operation, yellow for crashes, and various other colors for specific fault types like score changes or object misclassifications). Initial broad scans identify promising regions, followed by more constrained, focused scans to pinpoint specific fault-inducing parameters. The talk highlighted observing confidence score changes in 77.1% of cases and various false positives, demonstrating the NPU's susceptibility.
The most innovative technical contribution is the LLM-driven fault injection architecture. This system leverages an LLM (such as GPT) to manage the high-level strategy and optimization, while Keysight's internal software, FiPy, handles the low-level, high-frequency glitching operations. The architecture includes:
- Guardrails: Software-enforced limits on glitch parameters to prevent physical damage to the device.
- MCP (Model Context Protocol): Used for communication between the LLM and the physical setup.
- Supervisor GPT: This LLM's primary role is system health. It is given "eyes and ears" via a camera observing device LEDs, and "controls" through access to reset lines and power relays. It actively monitors for system crashes or unexpected states and can autonomously power cycle or reset the device to bring it back to a known good state. This is crucial for long-running experiments that might otherwise fail due to device instability.
- Optimizer GPT: Once the supervisor confirms the system is healthy, this LLM takes over to strategize the fault injection parameter search. It analyzes the results of previous glitches, identifies patterns, and suggests new parameter ranges (glitch length, voltage, timing) to explore, aiming to achieve a specific goal (e.g., finding a "red outcome" where a wrong PIN is accepted).
A striking example of the LLM's technical prowess was its ability to diagnose and fix a missing trigger issue. When the LLM, a Codex 5.5 model, noticed a discrepancy in the database (correct executions without corresponding triggers), it hypothesized that the trigger signal from the target device was too short for the fault injection hardware to reliably detect. Crucially, it was granted SSH access to the target device. The LLM then autonomously:
- Pulled down the device's binary.
- Reverse-engineered the binary to identify where the GPIO line was toggled.
- Modified the binary to extend the delay for the GPIO toggle, ensuring the trigger signal was long enough.
- Uploaded the modified binary back to the device.
This complex sequence of actions, involving code analysis, modification, and deployment, underscores the potential of LLMs to act as highly capable, autonomous security researchers, solving intricate hardware-software interaction problems that would typically require significant human expertise. The LLM then proceeded to perform more efficient parameter scans, focusing on the identified "transition boundary" where faults are most likely, significantly reducing the time to find desired outcomes compared to manual methods.
Demo / Proof of Concept
▶ Watch: LLM autonomously diagnoses and fixes hardware issues (18:00)
The talk effectively integrated several demonstrations and proof-of-concept scenarios to illustrate both the NPU vulnerabilities and the power of LLM-driven research.
The initial and most impactful visual demonstration involved a drone equipped with an object recognition system tracking a person and a car. Suddenly, after a simulated fault injection, the system's perception dramatically shifts, with the drone "seeing" a multitude of arbitrary objects—like "hot dogs everywhere"—and drawing bounding boxes around them. Crucially, the subsequent tracking algorithm then struggles, maintaining this state of confusion for an extended period, highlighting how a momentary NPU glitch can have prolonged, cascading effects on the overall AI system's behavior. This vividly showcased the real-world implications of confusing a CNN on an edge device.
A more controlled and detailed demonstration of the LLM-driven fault injection process was presented using a "Piñata" board, a simpler target for initial LLM training and testing. This demo focused on a basic pin code check application, where the goal was to achieve a "red outcome": sending an incorrect PIN that the system nevertheless accepts.
The demonstration utilized a custom Vibe coded UI that provided a live view of the LLM's activities. The UI displayed:
- Guardrails: Clearly showing the defined safe operational limits for glitch parameters.
- Live Fault Injection Attempts: A real-time visualization of each fault being injected and its immediate outcome (green for correct rejection, yellow for crash, orange for other, red for incorrect acceptance).
- Parameter Space Exploration: Graphs illustrating how the LLM was exploring the glitch length vs. voltage space, and the time instance of fault injection. This allowed the audience to observe the LLM's search strategy, initially broad, then becoming more focused as it learned.
- LLM Strategy Insights: The UI provided text output from the LLM, explaining its current strategy (e.g., "insufficient rows" during initial data collection) and rationale for parameter choices.
The demo successfully showed the LLM achieving the "red outcome" by iteratively adjusting parameters until the incorrect PIN was accepted. This demonstrated the LLM's ability to not only explore a vast parameter space but also to adapt its strategy based on observed outcomes, ultimately reaching the defined security goal.
Further proof of concept for the LLM's capabilities included:
- Supervisor Health Checks: The speaker explained how the Supervisor GPT, equipped with a camera observing the device's LEDs, could detect power failures (LEDs off) and autonomously trigger power cycles via connected relays, ensuring the experiment's continuity.
- Optimization Efficiency: Visual comparisons were shown between manual fault injection parameter searches (broad, less targeted) and the LLM's approach (highly focused on the "green-yellow transition boundary"), clearly demonstrating the LLM's superior efficiency in finding critical faults.
- Autonomous Debugging: The Codex 5.5 example, where the LLM diagnosed and fixed the missing trigger issue by reverse-engineering, modifying, and re-uploading firmware, served as a powerful proof of concept for the LLM's advanced problem-solving and system interaction capabilities beyond mere parameter tuning.
These demonstrations collectively showcased that physical fault injection on NPUs is a viable attack, and that LLMs can act as sophisticated, autonomous agents for hardware security research, significantly accelerating the discovery and analysis of such vulnerabilities.
Defensive Implications
▶ Watch: Q&A: Discussing NPU resilience and real-world impact (25:00)
The research presented by Jasper van Woudenberg carries significant implications for defenders working with edge AI devices and those integrating AI into their security workflows. The primary takeaway is that NPUs are not immune to physical attacks, and their vulnerabilities can lead to subtle yet critical failures in AI perception.
For developers and deployers of AI-powered edge devices, the findings necessitate a re-evaluation of security models. While software-level adversarial attacks receive considerable attention, physical fault injection directly targets the hardware, bypassing many traditional software defenses. Defenders should consider:
- NPU Resilience: How can NPUs be made more robust against voltage glitches or other physical manipulations? This could involve hardware-level countermeasures like voltage monitors, redundant computation, or error correction codes within the NPU itself. However, these are often costly and complex to implement.
- Layered Defenses in the AI Pipeline: The talk suggested adding "another AI to the mix" for validation. Instead of solely relying on the NPU's output, subsequent stages in the object detection and tracking pipeline should incorporate sanity checks. For instance, if an NPU suddenly reports "a thousand different things" or "the whole screen is filled with hot dogs," a higher-level validation module (which could itself be an AI or a rule-based system) should flag this as anomalous and potentially reject the NPU's output. This could involve checking for:
- Plausibility of Detections: Are the detected objects consistent with the environment? Is the number of detected objects suddenly spiking?
- Temporal Consistency: Does the tracking algorithm receive wildly inconsistent inputs from frame to frame?
- Confidence Thresholds with Anomaly Detection: Instead of just setting a static confidence threshold, dynamically adjust or trigger alerts when confidence scores behave erratically.
- Supply Chain Security: The ability to miniaturize fault injection circuits means these attacks could become a "supply chain problem." A small, covertly added "modchip" could inject faults into an otherwise legitimate edge AI device, enabling malicious actors to manipulate its perception. Defenders need to enhance their supply chain integrity checks, looking for unauthorized hardware modifications in addition to software tampering.
- Benchmarking AI in Security Tools: The research highlights the double-edged nature of AI: it's both a target and a powerful tool. For security teams looking to leverage LLMs or other AI models in their own defensive operations (e.g., for vulnerability research, incident response, or threat hunting), the importance of rigorous benchmarking cannot be overstated. As shown in the talk, the effectiveness and cost-efficiency of an LLM can vary wildly depending on the model's intelligence, the quality of instructions (toolkits, playbooks), and the specific task. Blindly adopting an LLM without empirical validation could lead to inefficient resource allocation or, worse, a false sense of security. Defenders should:
- Define Measurable Goals: Clearly articulate what success looks like for AI-driven security tasks.
- Test Multiple Models and Strategies: Don't assume one LLM or one set of instructions fits all.
- Quantify Performance and Cost: Understand the actual time, computational resources, and accuracy achieved by AI tools.
In summary, defenders must recognize the growing threat of physical attacks against AI hardware, implement multi-layered defenses that include anomaly detection at various stages of the AI pipeline, strengthen supply chain security, and adopt a data-driven, benchmarked approach when integrating AI into their own security operations.
Key Takeaways
- NPU Vulnerabilities are Real: Neural Processing Units (NPUs) are susceptible to physical fault injection attacks, which can reliably confuse Convolutional Neural Networks (CNNs) without crashing the device. This leads to critical misclassifications, false positives (e.g., "hot dogs everywhere"), and the ability to make objects "disappear" by manipulating confidence scores.
- LLMs Automate Complex Hardware Security Research: Large Language Models (LLMs) can act as powerful, autonomous agents for fault injection research. By serving as supervisors for system health and optimizers for parameter tuning, LLMs like GPTs can dramatically accelerate the discovery and analysis of hardware vulnerabilities.
- LLMs Exhibit Advanced Problem-Solving: Beyond mere parameter optimization, LLMs demonstrated capabilities like diagnosing setup issues, reverse-engineering binaries, and modifying device firmware autonomously, showcasing their potential for complex, system-level security tasks.
- Benchmarking AI Tools is Crucial: The effectiveness, efficiency, and cost of using LLMs in security research vary significantly based on the model's intelligence and the type/quality of instructions provided. Rigorous benchmarking is essential to ensure optimal results and avoid counterproductive strategies.
- AI is Both a Target and a Tool: As AI becomes pervasive in edge devices, it presents new attack surfaces for physical manipulation. Simultaneously, AI (specifically LLMs) offers innovative tools for security researchers to find and understand these vulnerabilities more efficiently.
- Layered Defenses for Edge AI are Necessary: Given the stability of glitched NPUs, relying solely on the NPU's output is insufficient. Defenders must implement additional validation layers, potentially using other AI or rule-based systems, to detect anomalous NPU behavior and ensure the overall integrity of AI-powered systems.
About the Speaker(s)
Jasper van Woudenberg is a distinguished expert in device security testing, currently contributing his expertise at Keysight Device Security Testing. His work focuses on understanding and mitigating vulnerabilities in embedded systems and hardware, with a particular emphasis on fault injection techniques. Known for his deep technical insights and analytical approach, Jasper's presentations often blend complex research with accessible explanations, sometimes even incorporating a well-timed "dad joke." His contributions are pivotal in advancing the security posture of modern computing hardware, especially in the burgeoning field of edge AI.
Reviews
Dr. Zero (Offensive Security Researcher) — MUST SEE
Jasper van Woudenberg took the industry's most abused buzzwords and actually built something terrifying. Banging voltage glitches into a Rockchip NPU to make a CNN hallucinate without crashing the SoC is solid, but unleashing Codex 5.5 to autonomously debug the GPIO trigger via SSH makes this mandatory viewing. This is what hardware research looks like when you strip away the marketing.
Heather Calloway (CISO) — SOLID
Keysight researchers demonstrated how physical voltage glitches can force edge AI systems—like cameras or autonomous vehicles—to hallucinate or ignore objects without crashing. More importantly, they used LLMs to fully automate the discovery of these hardware flaws. It's a highly technical hardware attack, but it forces a conversation about physical tampering and supply-chain integrity for any business deploying edge-compute fleets.