Evaluating Autonomous Vehicle Resilience

Evaluating Autonomous Vehicle Resilience

Speakers: Jiusheng Hu, Product Security Engineer, Zoox; Sanyam Gupta, Director, Zoox

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=kmPhJAz-5Rc

Reading time: 8 minutes

Type: Briefing

TL;DR

Zoox's product security team applied fuzzing techniques borrowed from software security to the teleoperation system of their autonomous robotaxi fleet. By generating over 50,000 mutated driving scenario variants from real-world data and running them in high-fidelity simulation, the team uncovered safety-critical edge cases — including collisions caused by a subtle bug in the AI's prediction module — that would have been impossible to find through manual testing.

Introduction

Autonomous vehicles are often presented as a binary proposition: either the AI drives alone, or a human takes over. The reality is more nuanced. For edge cases where the AI lacks confidence — a construction zone with shifted lane markings, an unusual obstacle — real-world robotaxi fleets rely on a human-in-the-loop system called teleoperation. A remote operator, watching live camera feeds and the AI's perception data, can provide waypoint suggestions that the vehicle follows while still maintaining full autonomous control of execution.

This collaboration sounds reassuring — and often is. But at Zoox, the product security team recognized an uncomfortable question: what happens when those teleoperation commands are wrong, whether through human error or adversarial manipulation? And more importantly, can we systematically find the scenarios where the AI fails to catch those bad commands before they cause harm?

Their answer was to adapt the most proven technique in software vulnerability research — fuzzing — to the physical world of autonomous driving.

How Teleoperation Works

▶ Watch: Teleoperation explained (02:00)

The teleoperation flow begins when a Zoox vehicle encounters a situation it cannot handle with full confidence. The vehicle sends a request — analogous to a student raising their hand — to a human operator at a remote operations center. The operator sees both a live camera feed and the AI's semantic view of the scene (detected vehicles shown as blue bounding boxes, planned trajectory shown as a dashed white line), then sends waypoint suggestions indicating a recommended path through the obstacle.

Critically, the AI remains in full control of vehicle actuation at all times. The operator's waypoints are advice, not commands. The vehicle interprets those waypoints through its full planning stack and can reject paths it determines to be unsafe. This architecture is designed to give the AI a confidence boost while preserving its safety guarantees.

The challenge is proving those safety guarantees hold. Standard validation involves defining a test scenario, replicating it in a test environment, providing it to a trained operator, and observing the outcome. If no collision occurs and the vehicle reaches its destination, the test passes. This works for nominal cases, but it cannot scale to the tail of possible scenarios — variations in waypoint positioning, vehicle speeds, pedestrian trajectories — where the most dangerous edge cases hide.

Applying Fuzzing to Autonomous Driving

▶ Watch: Fuzzing concept introduction (08:00)

In software security, fuzzing generates unexpected inputs to programs, monitors for crashes, and learns from results to generate progressively more interesting inputs. The Zoox team asked: why not apply the same paradigm to driving scenarios?

The architecture maps directly. The fuzzer sits at the center, generating driving scenario inputs rather than byte sequences. The target system is not a piece of software but the full teleoperation stack running inside Zoox's autonomous software, surrounded by traffic agents in a high-fidelity simulation. The monitor watches for safety violations — near misses, collisions — rather than program crashes.

Input generation begins with real-world driving data. Zoox vehicles collect millions of miles of logs, and the team extracted structured representations of teleoperation events from this corpus: the vehicle's position when a teleoperation command was sent, the type of command (waypoints, reverse, emergency stop), and the speed and trajectory of all traffic participants. These teleoperation parameters define the mutation space.

The fuzzer then applies programmatic offsets to these parameters. A recommended waypoint path might be shifted slightly forward, nudged toward the roadside, or extended into traffic. The mutations look plausible — close enough to what a well-trained operator might send — but differ enough to stress-test the AI's response to subtle errors. After each simulation run, the fuzzer uses the results to guide subsequent mutations: interesting behaviors (near misses, unexpected reactions) lead to similar variants, while boring outcomes push the fuzzer toward more aggressive exploration.

Discoveries: Collisions the Manual Process Would Never Have Found

▶ Watch: Collision case studies (14:00)

Starting from a real-world scenario database covering parking lot merges, double-parked vehicles, and intersections, the fuzzer generated over 50,000 mutated variants. Three collision cases illustrate the findings.

Case 1 — Merging from Parking: The scenario involves the vehicle merging into traffic from a parking spot following operator waypoints. Initial fuzzer mutations produced commands that the AI correctly rejected, stopping before a collision. But after many iterations of learning, the fuzzer found a waypoint configuration that looked nearly identical to a safe command but caused the vehicle to proceed into a mild collision with a parked car ahead. The AI did not reject the path.

Case 2 — Intersection Reverse: In this scenario, the vehicle was moving away from an intersection when the fuzzer injected a reverse command — a command the vehicle never requested. In baseline tests, the AI correctly detected the cross-traffic and waited. But through iterated mutation combining specific timing and path combinations for multiple agents simultaneously, the fuzzer triggered a subtle bug in the prediction module that caused the vehicle to move into the intersection just as another vehicle entered from the side, resulting in a side collision. Engineers analyzing the driving log confirmed this was a timing-sensitive interaction that no manual test would realistically produce.

▶ Watch: Root cause analysis and fix (22:01)

Case 3 — Intersection Turn Disruption: A right-turn maneuver was interrupted mid-execution by an emergency stop command, simulating a conflicting signal sent by a malicious or confused operator. After iterations, the fuzzer found a braking point that left the vehicle stopped in an intersection and unable to recover before a following vehicle caused a rear-end collision.

Closing the Loop: From Crash to Fix

Each collision, even a minor one, triggered a full analysis cycle. Engineers from the AI, perception, and planning teams examined driving logs to identify root causes. In the prediction module bug case, the fix involved retraining the decision logic; post-fix runs confirmed the AI could now correctly reject the unsafe waypoint portion and navigate around it.

Importantly, the team emphasized that fixes are evaluated not just for the specific scenario that revealed the bug, but for the broader class of similar scenarios. The goal is not to patch individual test cases but to harden the AI's general decision-making.

The fuzzing framework is now expanding beyond teleoperation commands. The team is applying similar techniques to the perception module — injecting rare and distorted sensor inputs to test whether downstream modules respond safely — and to the planning module itself, stress-testing decision-making logic under uncommon, high-severity conditions like aggressive cut-ins.

Notable Quotes

"Safety is not just about preparing for the best under predictable conditions. We also need to prepare for the worst."

— Jiusheng Hu ▶ 08:00

"The fuzzer finally found a suggestion that can slip right under the radar. It looks very similar to the previous one, but this time the vehicle moves, it didn't yield, and finally a mild collision happened."

— Jiusheng Hu ▶ 16:00

"This is a subtle bug sitting in the prediction module, and this bug can only be triggered under a very specific combination of timing and paths of both or multiple agents at the same time. Imagine we only tested cases manually without the fuzzer — this kind of subtle bug might never have been found."

— Jiusheng Hu ▶ 18:00

"The safety and security of the autonomous vehicle must be able to scale. This requires more than just nominal testing — it's also about actively challenging the system just like a real-world attacker would."

— Jiusheng Hu ▶ 24:01

Key Takeaways

• Fuzzing scales safety testing for autonomous vehicles in ways manual testing cannot. Generating 50,000+ scenario variants from real-world data surfaced edge cases with specific timing and trajectory combinations that no human test team would design.
• Teleoperation is both a safety feature and an attack surface. Commands that look like reasonable operator input can, in certain configurations, bypass AI safety checks — whether due to benign mistakes or adversarial intent.
• The prediction module is a critical failure point. The intersection collision case revealed that bugs triggered only by specific multi-agent timing combinations can be hiding in the prediction stack, undetected by nominal testing.
• The fix cycle must generalize, not just patch. After each discovered collision, the team retrained decision logic to cover the broader class of similar scenarios, not just the specific mutated variant that caused the failure.
• The methodology extends beyond teleoperation. The same fuzzing approach is now being applied to perception (sensor input distortion) and planning (high-severity edge case scenarios), suggesting a framework for comprehensive AV security validation.

Slides were not listed as available for this talk.