Side-channel Inference of User Activities in AR/VR Using GPU Profiling

Side-channel Inference of User Activities in AR/VR Using GPU Profiling

Speakers: Seonghun Son

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=f8AFeMEoxZE

Overview

As AR/VR headsets move beyond entertainment into medical, education, and industrial applications, the privacy implications of these always-on immersive devices become critical. This research reveals that the built-in GPU profiler on MetaQuest devices -- the dominant XR platform with over 80% market share -- creates a measurable side-channel that allows a background process to infer user activities with alarming accuracy. Using just a 1 Hz sampling rate (one data point per second), the researchers achieved nearly 100% accuracy in identifying which applications and websites a user is viewing, over 80% accuracy in identifying specific 3D objects being rendered, and 100% accuracy in counting meeting participants.

The attack requires no special SDK, no concurrent app execution, and operates at a sampling rate so low it is difficult to detect. The researchers disclosed their findings to Meta's security team and received a Meta bug bounty award, confirming the real-world impact of the vulnerability.

Background

▶ Watch: Motivation: GPU profiling as a side-channel in AR/VR devices (0:00)

AR/VR devices rely heavily on GPU resources to render 3D environments in real-time, making GPU utilization patterns a rich source of information about what the user is experiencing. MetaQuest devices, built on the Qualcomm Snapdragon XR Gen 2 chip, include a built-in GPU profiler called OVR GPU Profiler that can be accessed by third-party applications. This profiler exists for legitimate performance optimization purposes -- identifying GPU bottlenecks to improve rendering quality -- but it also creates an unintended side-channel.

Previous GPU side-channel research typically required high-resolution sampling rates and concurrent application execution, making attacks detectable and easier to block. Earlier mitigations proposed lowering GPU counter resolution or blocking concurrent foreground applications. This research demonstrates that these mitigations are insufficient: the attack works at an extremely low 1 Hz sampling rate, runs as a background script rather than a concurrent app, and still achieves high classification accuracy across multiple attack scenarios.

Key Findings

▶ Watch: System pipeline: fingerprint collection, reverse engineering, classification (4:00)

The researchers identified 30 GPU metrics (out of 72 on Quest 2 and 78 on Quest 3) that correlate with 3D object rendering and can reveal user activities:

Application Fingerprinting: By profiling GPU metrics for 30 seconds, the attack can distinguish between 100 different standalone apps from the Meta Quest app store and 100 different WebXR websites with nearly 100% accuracy. Each application produces a distinctive GPU usage pattern that serves as a reliable fingerprint.

3D Object Identification: Within a running application, the attack can identify which of 35 different 3D objects are being rendered with over 80% accuracy by analyzing metrics like the non-base level texture percentage, which measures how often the GPU uses smaller copies of textures as objects move closer or farther from the user.

Meeting Participant Counting: In virtual meeting scenarios, the GPU fingerprints show a distinctive step behavior as participants are added, enabling 100% accuracy in counting meeting participants using any of 20 individual metrics with a random forest model.

Open-World Validation: Testing with Meta's pre-installed Layout app and five recruited users (with IRB approval from Iowa State University), the attack achieved over 80% accuracy in both static and dynamic real-world scenarios for identifying which furniture items users were placing in their AR/VR environment.

Technical Deep Dive

▶ Watch: Results: 100% app classification, 80%+ 3D object identification (8:00)

The attack pipeline operates in three stages:

Stage 1 - Data Collection: A normal-looking malicious app triggers a background script before terminating. The script continuously profiles GPU metrics using the built-in OVR GPU profiler at 1 Hz while the user performs normal activities in the foreground. Collected traces are saved to the device for later exfiltration.

Stage 2 - Feature Engineering: The researchers performed reverse engineering of collected fingerprints to select the most informative metrics. A key metric is non-base level texture percentage, which tracks how often the GPU samples smaller texture copies (mipmaps). When a 3D object moves closer and becomes larger on screen, the GPU samples more mipmap levels, increasing this percentage. When objects recede, the percentage drops. The researchers conducted correlation analysis between pixel counts and GPU metrics across both Quest 2 and Quest 3 devices in AR and VR modes, confirming strong correlations. Dynamic correlation analysis also showed that object movement speed impacts GPU fingerprinting patterns.

Stage 3 - Classification: The researchers applied standard machine learning models including CNN, LSTM, Random Forest, and SVM to classify user activities. The models take GPU metric time series as input and output predictions about which app is running, what objects are being rendered, or how many participants are in a meeting.

The practical validation using Meta's Layout app tested two scenarios: a static scenario where participants generated one of five furniture objects for 5 seconds (100 seconds total per item), and a dynamic scenario where participants freely dragged furniture in and out of their field of view for 40 seconds (repeated five times per object).

Demo / Proof of Concept

▶ Watch: Open-world validation with Meta Layout app and real users (10:00)

The researchers demonstrated their attack using Meta's built-in Layout app, which comes pre-installed on all MetaQuest devices. In the demo, users select furniture or 3D objects from an in-scene panel, place them into their field of view, and interact with them through moving, resizing, and rotating. The background GPU profiling script captures the distinctive GPU fingerprints generated by each piece of furniture, enabling identification of which objects the user is placing and interacting with.

The attack was validated across both Quest 2 and Quest 3 hardware, and also tested on Microsoft HoloLens, though the presentation focused on MetaQuest due to its dominant market share. The researchers confirmed they received a Meta bug bounty award for the disclosed vulnerabilities.

Defensive Implications

▶ Watch: Meta bug bounty award and Q&A on new app generalization (14:00)

The researchers propose three categories of mitigation:

Restrict GPU Profiler Access: GPU profiler APIs are currently gated behind developer mode and app store approval, meaning normal apps cannot sample these metrics. However, the research demonstrates that a malicious app can trigger a background script that continues profiling after the app terminates, bypassing this restriction.

Dynamic Detection: App developers or platform vendors could detect repetitive profiling polling patterns and either warn the user or throttle access with minimal overhead. This would catch the continuous 1 Hz sampling pattern used in the attack.

Noise Injection: Adding controlled dummy GPU workloads or random rendering operations could blur the GPU fingerprints, making classification more difficult. However, this approach introduces performance overhead, which is particularly problematic for latency-sensitive XR applications where frame drops cause motion sickness.

The broader implication is that any hardware profiling interface accessible to applications creates a potential side-channel in immersive computing environments. As XR devices move into enterprise settings for remote collaboration, training, and design review, the ability to infer meeting participant counts and specific objects being viewed raises serious confidentiality concerns.

Key Takeaways

• Built-in GPU profilers on MetaQuest devices create a measurable privacy side-channel that works at just 1 Hz sampling rate
• A background script can fingerprint applications with nearly 100% accuracy, identify 3D objects at 80%+ accuracy, and count meeting participants at 100% accuracy
• The attack requires no special SDK, no concurrent app execution, and operates below typical detection thresholds
• Non-base level texture percentage and 29 other GPU metrics correlate strongly with rendered content across both Quest 2 and Quest 3
• Real-world validation with five users and Meta's pre-installed Layout app confirmed over 80% accuracy in practical scenarios
• Meta acknowledged the findings with a bug bounty award; proposed mitigations include restricting profiler access, detection of polling patterns, and noise injection

About the Speaker(s)

Seonghun Son (presenting as Tongon) is a researcher from Iowa State University working on security and privacy in emerging computing platforms. This research was conducted in collaboration with Purdue University and the American University of Sharjah. The team's work focuses on side-channel attacks in XR environments, and their responsible disclosure to Meta resulted in a bounty award recognizing the practical impact of their findings.