Beyond Fun and Games: Understanding the Threat Landscape of Extended Reality

Beyond Fun and Games: Understanding the Threat Landscape of Extended Reality

Speakers: József Ottucsák, Security Architect

Conference: Hacktivity

YouTube: https://www.youtube.com/watch?v=zbmufA4BzBk

Overview

In his Hacktivity talk, "Beyond Fun and Games: Understanding the Threat Landscape of Extended Reality," Security Architect József Ottucsák delivered a crucial overview of Extended Reality (XR), moving beyond its popular perception as a consumer gaming technology. Ottucsák emphasized that while XR is still in its nascent stages, akin to early mobile phones, its rapid evolution and increasing adoption across critical industries necessitate a proactive approach to security. The core message of the presentation is that XR systems are fundamentally cyber-physical systems, meaning their digital interactions have direct and tangible consequences in the real world, a characteristic that elevates their security risk profile far beyond traditional IT systems.

Ottucsák highlighted the current media narrative, which often frames XR from a purely consumer perspective, overlooking its profound implications for industrial and enterprise applications. From military training and healthcare to manufacturing and remote support, XR is poised to revolutionize various sectors where safety, data integrity, and operational continuity are paramount. By drawing parallels with Industrial Control Systems (ICS), the talk underscored the critical need for security professionals to engage with XR technology now, rather than waiting for it to become a mainstream staple, ensuring that robust security frameworks are integrated from its foundational development.

The presentation aimed to demystify XR by distinguishing it from Augmented Reality (AR), Mixed Reality (MR), and Virtual Reality (VR), defining it as a comprehensive continuum. Ottucsák provided a technical overview of common XR devices, detailed the unique threats they introduce, and outlined existing and emerging security controls. His call to action was clear: to foster a community of security engineers and professionals who can anticipate and mitigate the complex risks associated with these cutting-edge, real-world interacting technologies, ensuring a secure and reliable future for extended reality.

Background

▶ Watch: Introduction and why XR security matters (0:00)

Extended Reality (XR) represents a technological frontier that, while often depicted as a present-day phenomenon by the media, is still very much in its infancy. József Ottucsák noted that the technology faces significant bottlenecks in computational power and battery life, placing it at a stage comparable to the bulky mobile phones of decades past. This early phase, however, presents a unique opportunity for security professionals to shape its future rather than reacting to entrenched vulnerabilities.

The core distinction Ottucsák made is between the media's consumer-centric view of XR and its critical applications in industrial and enterprise settings. In these contexts, XR transcends mere entertainment, becoming integral to operations where security and safety carry profound real-world consequences. This is because XR systems are inherently cyber-physical systems: digital actions and data flows directly influence or interact with the physical environment. Unlike abstract data manipulation on a remote server, an XR system error or compromise can lead to physical harm, operational disruption, or severe data breaches impacting tangible assets and human lives. This characteristic places XR in a similar risk category to Industrial Control Systems (ICS) and other business-critical infrastructures.

To clarify the terminology, Ottucsák defined XR as an overarching continuum encompassing AR, MR, and VR:

• Augmented Reality (AR): Enriches the real physical world with digital information (e.g., a phone app overlaying furniture in a room).
• Mixed Reality (MR): Gradually replaces more and more of the physical world with virtual elements, allowing for interaction between real and virtual objects.
• Virtual Reality (VR): Aims to completely replace the user's perception of the physical world with an entirely virtual environment.

The speaker emphasized that the lines between these categories are increasingly blurred, making a strict distinction challenging, hence the utility of the "Extended Reality" umbrella term.

Common XR devices extend beyond the typical head-mounted displays (HMDs) to include everyday smartphones equipped with depth sensors, spatial displays that render 3D images without glasses, and caves – large physical spaces with multiple projectors that create immersive environments. However, Head-Mounted Displays (HMDs) like the HoloLens, Apple Vision Pro, and Meta Quest/Rift devices are becoming the most prominent and flexible form factor.

The industrial applications of XR are vast and impactful:

• US Air Force: Utilizes VR for pilot training, significantly reducing costs.
• Retail: Companies like Ikea offer AR apps to visualize 3D furniture in homes, while Amazon allows virtual shoe try-ons.
• Healthcare: XR is employed for surgical training, augmenting patient information for doctors, and potentially enabling remote assistance for complex procedures, mitigating risks associated with traditional, high-stakes surgeries.
• Manufacturing, R&D, Remote Support: XR facilitates prototyping, design collaboration, and remote guidance for technicians, allowing a support engineer to visually guide a customer through troubleshooting a physical device. These use cases highlight the critical need for security, as intellectual property theft or operational sabotage through compromised XR devices could have devastating economic and strategic impacts.

Key Findings

▶ Watch: Diverse industry applications of XR beyond gaming (5:50)

The talk identified several critical findings regarding the XR threat landscape, shifting the focus from mere consumer entertainment to enterprise-grade security concerns.

Firstly, XR devices, particularly Head-Mounted Displays (HMDs), are essentially mobile devices on steroids. They are typically powered by ARM-based Systems-on-Chip (SoCs), such as Qualcomm's DXR series (found in Meta Quest and HoloLens) or Apple's M2 chip (in the Vision Pro). Crucially, these devices run forks of existing, well-established operating systems: Meta Quest uses a modified Android Open Source Project (AOSP), Microsoft's HoloLens runs Windows Holographic Edition, and Apple's Vision Pro utilizes Vision OS, likely derived from iOS/macOS. While this inheritance provides some baseline security features, it also means they are susceptible to many of the same vulnerabilities found in their mobile counterparts, alongside new, XR-specific attack vectors.

Secondly, XR devices are prolific data collectors, gathering an unprecedented volume and type of sensitive information from both the user and their physical environment. This includes depth data and point clouds (3D mapped images of the surroundings), high-resolution video and audio streams, biometric data (eye, head, and hand tracking), network credentials, sensitive application data, and extensive telemetry on user behavior and device usage. The ability to map an entire room, listen to conversations, and track precise movements means that a compromised XR device can serve as a potent surveillance tool, extending its reach beyond the user to anyone else in the same physical space.

Thirdly, the inherent cyber-physical interaction of XR systems introduces unique and severe risks that go beyond traditional information security. The possibility of physical damage to the user or environment is a tangible threat; for instance, a malicious actor could disable safety features or alter the user's perception of their surroundings, leading to accidental collisions or injuries. The concept of altering user senses to trick them into taking actions is akin to clickjacking but applied to the physical world, where manipulated virtual overlays could induce users to interact with real objects in unintended or dangerous ways.

Finally, while some security controls are emerging, the landscape is uneven. Consumer-grade XR devices often rely on weaker authentication methods like PINs, which are susceptible to shoulder surfing or inference based on hand movements. Enterprise-ready devices, such as the HoloLens 2, incorporate more robust features like 5-2 biometric authentication and integrate with Mobile Device Management (MDM) solutions. However, the prevalence of sideloading applications, the potential for supply chain risks in application development (Unreal Engine, Unity, OpenXR), and the constant emergence of application and OS vulnerabilities mean that a comprehensive defense strategy is paramount. The interaction with other connected devices (e.g., robots, industrial machinery) further complicates the threat model, as a compromise in the XR interface could cascade into control over critical physical systems.

Technical Deep Dive

▶ Watch: Technical overview of HMDs and their data collection (8:00)

The technical architecture of Head-Mounted Displays (HMDs), the most prevalent form of XR device, reveals them to be sophisticated, sensor-rich computing platforms. At their core is a VR console, which encompasses the device's runtime, operating system, and applications, all designed to interface with a multitude of physical sensors and drive the electronic displays. Most HMDs leverage ARM-based Systems-on-Chip (SoCs), specifically Qualcomm's DXR series for devices like the Meta Quest and HoloLens, and Apple's M2 chip for the Vision Pro.

The operating systems powering these devices are typically forks of established platforms, allowing manufacturers to leverage existing security hardening and development ecosystems. Meta has transitioned its Quest line to a modified Android Open Source Project (AOSP), while Microsoft's HoloLens runs on Windows Holographic Edition, and Apple's Vision Pro uses Vision OS, likely based on its iOS/macOS lineage. This foundational software stack is responsible for managing the complex interplay between hardware, sensors, and applications.

A critical aspect of HMDs is their extensive array of physical sensors. These include:

• Eye tracking: Monitors gaze direction and pupil movement.
• Head tracking: Records head orientation and position in 3D space.
• Hand tracking: Captures precise hand and finger movements for interaction.
• Gyroscopes and Accelerometers: Provide inertial data for motion sensing.
• Cameras: Crucially, these are not just for flat images but also collect depth data and point clouds, enabling the device to construct a detailed 3D map of the user's environment. This capability allows for sophisticated environmental understanding and interaction, but also represents a significant privacy and security risk.

These sensors feed data into the device, which then processes it to render virtual content on the electronic displays (often two per eye) and facilitate user interaction. The devices also incorporate standard network interfaces such as Wi-Fi, Bluetooth, and USB, which serve as pathways for data transmission and potential attack vectors.

The data gathered by XR devices is exceptionally sensitive and diverse:

• Environmental Data: Point cloud data (3D mapped images), video, and audio streams of the physical environment. This can include mapping a user's entire home or office, and listening to conversations.
• User Biometrics and Behavior: Eye, head, and hand tracking data, which can infer attention, emotional states, and physical actions. Telemetry information records usage habits, device interactions, and metadata about user activity.
• Application and System Data: Network credentials, application secrets, and other sensitive information processed by untrusted applications.

Applications for XR devices are typically developed using popular engines like Unreal Engine, Unity, and OpenXR. This introduces common software development risks, including supply chain vulnerabilities (from third-party libraries and SDKs), bad coding practices, and the potential for malicious or poorly secured applications to be distributed. While marketplaces offer some sanity checks, sideloading applications bypasses these controls, opening the door for untrusted code execution.

The overall threat model for XR devices places the XR device itself at the center. It connects to the internet via 5G, Wi-Fi, or USB, crossing trust boundaries to send data to various external entities:

• Application Servers: For functionality and content delivery.
• Device Manufacturers: For telemetry, updates, and service provision.
• Third-Party Data Processors: Often cloud providers like AWS, Azure, or GCP, where sensitive user and environmental data inevitably resides.

Crucially, the interaction with the physical environment, the user, and connected devices fundamentally alters the threat landscape. A compromised XR device can lead to:

• Physical Harm: If safety measures are disabled, or the user's perception is altered, causing them to collide with objects or people.
• Information Leakage: Voice commands are spoken aloud, and grand physical gestures for interaction can be observed or inferred, making privacy challenging in shared physical spaces.
• Control over Critical Systems: In enterprise settings, XR devices may interface with robots, industrial machinery, or even critical infrastructure (e.g., surgery robots, aircraft systems). A security breach in the XR interface could thus extend to control over these real-world devices, with potentially catastrophic consequences. This highlights the need for robust isolation and authentication mechanisms when integrating XR with operational technology.

Demo / Proof of Concept

▶ Watch: Summary of critical XR security threats (16:00)

The talk focused on a comprehensive analysis of the XR threat landscape and defensive strategies, rather than presenting a live demonstration or proof of concept of an exploit. József Ottucsák's presentation was structured as an informative overview designed to raise awareness and provide a foundational understanding of XR security.

Defensive Implications

▶ Watch: Existing security controls and enterprise deployment advice (18:00)

Understanding the unique threat landscape of Extended Reality necessitates a robust and multi-faceted defensive strategy, extending beyond traditional IT security. Defenders must consider both the digital and physical repercussions of a compromise.

Firstly, physical safety precautions are paramount. Users must be educated on the risks of operating XR devices in unsafe environments, and enterprise deployments should implement clear spatial awareness guidelines and potentially physical barriers. From a technical perspective, robust integrity checks on safety measures within the XR operating system and applications are crucial to prevent malicious alteration of user perception or disabling of guardian systems.

Secondly, addressing the threat of altered user senses requires vigilance against sophisticated forms of clickjacking or perception manipulation. This involves ensuring the authenticity and integrity of virtual overlays, preventing unauthorized injection of content, and providing clear indicators when the virtual environment is being modified or overridden. Secure rendering pipelines and application sandboxing are vital here.

Thirdly, the extensive surveillance capabilities of compromised XR devices demand stringent access controls and data protection. If an attacker gains rootkit-level access to an HMD, they can effectively see and hear everything in the user's environment. This necessitates:

• Strong application sandboxing: To limit an application's access to sensor data.
• Granular permission models: Allowing users to control what data is accessed by which application, similar to modern mobile OS.
• Encryption of sensor data at rest and in transit: Protecting sensitive point cloud, video, and audio data.
• Regular security audits: Especially for applications handling sensitive environmental data.

Fourthly, the issue of weaker security controls on many consumer devices, such as reliance on PINs, highlights the need for stronger authentication. Enterprise deployments should mandate 5-2 biometric authentication (e.g., eye-tracking-based authentication, facial recognition) where available, and integrate with Mobile Device Management (MDM) solutions. MDM allows organizations to enforce security policies, manage applications, restrict sideloading, and remotely wipe devices, providing a critical layer of control in corporate environments. The Microsoft HoloLens 2 deployment guide is specifically recommended as a resource for establishing enterprise-grade security for XR devices.

Fifthly, addressing application and OS vulnerabilities requires a proactive approach:

• Secure Development Lifecycle (SDL): For in-house XR application development, incorporating security from design to deployment.
• Vulnerability Management: Regular patching and updates for the XR operating system and applications.
• Supply Chain Security: Thorough vetting of third-party libraries, SDKs, and components used in XR development (Unreal Engine, Unity, OpenXR).
• Code Signing Protections: Ensuring that only trusted, signed applications can be installed and executed, preventing the installation of malicious software.
• Locked Bootloaders: Preventing unauthorized flashing of custom or rooted firmware onto devices.

Finally, organizations must account for data leakage and the interaction with connected devices. Implementing network segmentation can isolate XR devices from critical operational technology. Data loss prevention (DLP) strategies should be extended to cover the unique data types generated by XR (e.g., point clouds). When XR devices interface with robots or industrial systems, robust authentication and authorization mechanisms are crucial to ensure that only authorized commands are executed, preventing a compromise from cascading into physical control.

Ottucsák also highlighted the importance of leveraging existing security features. The fact that XR operating systems are built upon established platforms (Android AOSP, Windows Holographic, iOS-derived Vision OS) means they inherit years of OS hardening efforts. Security teams should understand and maximize these built-in controls. Learning from industry leaders, the speaker recommended reviewing the Meta Quest 2 "Defense through Offense" write-up, which details how Meta's engineering team proactively identifies and fixes memory safety bugs through internal exploitation, offering valuable insights into robust security practices.

Key Takeaways

• XR is a Cyber-Physical System: Unlike traditional IT, Extended Reality directly interacts with and impacts the physical world, introducing unique and severe risks that necessitate a security approach akin to Industrial Control Systems.
• Pervasive Sensitive Data Collection: XR devices gather highly sensitive data, including 3D environmental maps (point clouds), video, audio, biometrics, and extensive usage telemetry, making them potent tools for surveillance if compromised.
• Unique Threats Beyond Traditional IT: Beyond standard software vulnerabilities, XR introduces risks of physical harm to users or environments, manipulation of user senses (physical-world clickjacking), and widespread environmental surveillance.
• Enterprise Adoption Demands Robust Controls: For industrial and enterprise use cases, basic security measures like PINs are insufficient. Organizations must implement strong authentication (biometrics), Mobile Device Management (MDM), locked bootloaders, and rigorous code signing.
• Proactive Engagement is Crucial: Security professionals need to engage with XR technology now, understanding its architecture, vulnerabilities, and defensive strategies, rather than waiting for it to become fully mature and widely deployed.
• Leverage Existing Hardening and Learn from Industry: Built-in OS hardening from underlying platforms (Android, Windows, iOS) provides a foundation, but continuous learning from industry best practices, such as Meta's "Defense through Offense" for memory safety, is essential.

About the Speaker(s)

József Ottucsák is a Security Architect with a deep passion for cutting-edge technology. With over 10 years of experience in the tech industry, he identifies as a hacker, early adopter, and tech evangelist. His enthusiasm for tinkering with computers, consoles, and other technical gadgets dates back to his early childhood, starting at the age of six. He brings this extensive background and curiosity to his work, exploring the security implications of emerging technologies like Extended Reality.