A primer on security and safety in eXtended / Augmented Reality environments (XR)i

Kavya Pearlman (Founder · XR Safety Initiative)

Swiss Cyber Storm 2021 · Day 1 · Scenario_2

Overview

In her compelling talk at Swiss Cyber Storm, Kavya Pearlman, founder of the XR Safety Initiative (XRSI), delivered a critical primer on the often-overlooked security, safety, and privacy implications of eXtended Reality (XR) environments and the emerging Metaverse. Pearlman's central thesis is that the world is on the cusp of, if not already experiencing, a profound "paradigm shift" driven by these immersive technologies, leading to an era of "constant reality capture." This shift, she argues, will generate unprecedented volumes of highly sensitive biometric and behavioral data, presenting a double-edged sword: immense potential for good, but also severe risks if not proactively managed through robust standards, ethics, and regulation.

Watch on YouTube · Slides

Visual summary for A primer on security and safety in eXtended / Augmented Reality environments (XR)i by Kavya Pearlman
Visual summary for A primer on security and safety in eXtended / Augmented Reality environments (XR)i by Kavya Pearlman

Key moments

  1. 1:15 XRSI's mission for safe, inclusive emerging tech
  2. 4:30 The history of data weaponization and real-world impact
  3. 8:00 Understanding the metaverse: connect, create, commerce
  4. 18:15 Stanford research: 2 million data points in 20 minutes VR
  5. 22:00 Biometrically inferred data: eyes, gait, spatial audio
  6. 27:45 Specific cyber attacks: tracker, human joystick, ransomware
  7. 33:10 Metaverse: broader problem beyond just XR devices
  8. 35:30 XR's 'two-way sword': healing vs. weaponizing mental health

A Primer on Security and Safety in eXtended / Augmented Reality Environments (XR)

Speakers: Kavya Pearlman, Founder, XR Safety Initiative

Conference: Swiss Cyber Storm

YouTube: https://www.youtube.com/watch?v=x3hWKCallGs

Overview

In her compelling talk at Swiss Cyber Storm, Kavya Pearlman, founder of the XR Safety Initiative (XRSI), delivered a critical primer on the often-overlooked security, safety, and privacy implications of eXtended Reality (XR) environments and the emerging Metaverse. Pearlman's central thesis is that the world is on the cusp of, if not already experiencing, a profound "paradigm shift" driven by these immersive technologies, leading to an era of "constant reality capture." This shift, she argues, will generate unprecedented volumes of highly sensitive biometric and behavioral data, presenting a double-edged sword: immense potential for good, but also severe risks if not proactively managed through robust standards, ethics, and regulation.

Pearlman, drawing on her extensive background in security advising for major tech companies and virtual worlds, frames the discussion not merely as a technical challenge but as a societal imperative. She highlights how historical abuses of data in social media have led to real-world consequences, from undermining democracies to fueling insurrections. The Metaverse, with its deeper immersion and pervasive data collection, magnifies these risks, shifting the "attack surface" from traditional networks to individuals' "living rooms and brains." XRSI's mission, as Pearlman outlines, is to bring together experts to establish a common language and develop frameworks that ensure these powerful technologies develop safely, securely, and ethically.

Background

▶ Watch: XRSI's mission for safe, inclusive emerging tech (1:15)

Kavya Pearlman's journey into XR safety began with a firsthand observation of the trade-offs between technological advancement and individual privacy. Her experience in 2016 advising Facebook on third-party security risks, followed by her role as Head of Security for Linden Lab (creators of the pioneering virtual world Second Life), provided a front-row seat to the evolving landscape of digital interaction and its inherent vulnerabilities. This historical perspective forms the bedrock of her argument: that the weaponization of information, though not new, is reaching an unprecedented scale with XR.

Pearlman meticulously traces the trajectory of data misuse, starting with the 2011 Egyptian Revolution, where social media facilitated regime change, prompting adversarial entities to "take notes." This was followed by the 2012 US presidential election, which demonstrated the power of digital media campaigns and data utilization. The subsequent Brexit referendum and the infamous Cambridge Analytica scandal revealed the capacity for data-driven polarization and manipulation, with Alexander Nix, former CEO of Cambridge Analytica, famously boasting of collecting "four or five thousand data points on every adult in the United States" to enable "ever increasingly targeted" communication. Pearlman starkly connects these events to the January 6th insurrection, illustrating how weaponized information can influence individuals to the extent of undermining democratic institutions.

Against this backdrop, Pearlman introduces eXtended Reality (XR) as an umbrella term encompassing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). VR immerses users in fully digital environments via head-mounted displays (HMDs), utilizing inside-out or outside-in tracking. AR overlays digital objects onto the real world, familiar through social media filters or devices like Facebook's Ray-Ban Stories glasses. MR allows manipulation of real-life objects using HMDs, such as controlling robots. XRSI has actively worked to standardize these terminologies to foster a common understanding.

The discussion then pivots to the Metaverse, a concept popularized by Mark Zuckerberg's declaration of an intent to build a "metaverse company." Pearlman clarifies that the Metaverse is not merely a larger virtual world like Second Life (which she terms a "multiverse"), but a foundational "alternate spatial layer" that will integrate a multitude of technologies. Its anatomy, as researched by XRSI and presented to NASA's STAR standardizing body, involves seamless connectivity, social interaction, creation of digital assets, and robust virtual economies (e.g., Linden Dollars, NFTs like Bored Apes). This future will be powered by advanced infrastructure, including AR glasses, 5G/6G networks, and pervasive Artificial Intelligence (AI), all contributing to an unprecedented level of data capture. Pearlman refers to this as the "Collingwood Dilemma"—the inherent difficulty in quantifying technology's impact without collecting data for its development, by which point it may be too late to reverse negative consequences, particularly with self-learning AI algorithms.

Key Findings

▶ Watch: Understanding the metaverse: connect, create, commerce (8:00)

The central and most alarming finding presented by Kavya Pearlman is that the advent of XR and the Metaverse ushers in an "era of constant reality capture," fundamentally redefining the scope and sensitivity of personal data collection. This is not merely about traditional online data but about deeply intimate biometric and behavioral information. Pearlman cites a critical 2018 Stanford University research finding that "in 20 minutes you can get over 2 million unique data points on an individual in VR." This figure dramatically overshadows the 5,000 data points cited by Cambridge Analytica for social media, underscoring the exponential increase in surveillance potential.

This pervasive data collection leads to a critical shift in the attack surface. Historically, cybersecurity focused on "nodes and servers and the network." In the XR era, the attack surface expands dramatically to "our living room and to our brain." This implies that vulnerabilities are no longer confined to digital infrastructure but extend to the physical spaces users occupy and even their cognitive processes.

Pearlman then unveils specific, novel cyberattacks identified by a team led by Ibrahim Bajilli, a senior advisor at XRSI and Senior Assistant Dean at the University of New Haven, in their 2019 research:

  1. Tracker Attack: Attackers can compromise a machine tethered to an XR device, hijack its front-facing camera, and stream real-time video of the user's physical environment, allowing them to "look inside the person's room." This poses a significant physical privacy risk, especially for "persons of interest."
  2. Chaperone Attack: This targets the virtual safety boundaries (known as "chaperones") that users set in VR to prevent collisions with real-world objects. Attackers can manipulate these boundaries, expanding or shrinking them at will.
  3. Human Joystick Attack: Building on the chaperone attack, this allows attackers to surreptitiously move a user across their physical room without their knowledge. While seemingly innocuous in a game, Pearlman warns of its potential for targeting individuals or disrupting critical activities.
  4. Ransomware Attack: Attackers could take over the entire display of an XR device, rendering it unusable until a ransom is paid, akin to traditional ransomware but with a far more immersive and disruptive impact.

A profound concern raised by Pearlman is the current regulatory vacuum. Despite the immense potential for misuse of this deeply personal data—such as denying insurance coverage based on "pre-existing conditions" inferred from biometric data—there are "no laws, there are not even common understandings yet." This lack of a legal and ethical framework leaves individuals highly vulnerable in this rapidly evolving technological landscape.

Technical Deep Dive

▶ Watch: Biometrically inferred data: eyes, gait, spatial audio (22:00)

The technical underpinnings of XR and the Metaverse revolve around the unprecedented collection and convergence of diverse data types. Pearlman meticulously details how this new reality is being constructed and the specific data points at risk.

XR and Metaverse Anatomy:

As defined by XRSI, eXtended Reality (XR) serves as an umbrella term for Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). VR, exemplified by head-mounted displays (HMDs), creates fully immersive digital environments, relying on precise inside-out or outside-in tracking to map user movements, down to individual finger gestures. AR overlays digital information onto the real world, seen in ubiquitous social media filters or devices like Facebook's Ray-Ban Stories smart glasses, which can record up to 30 seconds of video. MR goes a step further, enabling interaction and manipulation of real-world objects using HMDs, such as controlling a robotic arm.

The Metaverse is envisioned as a seamless, persistent, and interconnected digital layer that transcends these individual XR modalities. Its core functions include facilitating connection and socialization in virtual spaces, enabling creative expression (e.g., building a virtual Taj Mahal in minutes for cybersecurity awareness), and supporting burgeoning virtual economies through Linden Dollars (from Second Life) and Non-Fungible Tokens (NFTs). This requires a robust infrastructure powered by high-bandwidth, low-latency 5G and 6G networks, and ubiquitous Artificial Intelligence (AI) embedded directly into devices like smart glasses. These AI systems will constantly learn user preferences, behaviors, and even emotional states, transforming the user experience into one of "constant reality capture."

The New Data Landscape:

Beyond traditional demographic and preference data (like the "5,000 data points" of Cambridge Analytica), XR introduces a new tier of highly sensitive information:

  • Biometrically Inferred Data: Pearlman emphasizes that the "eyes are not the only prize," but they are significant. Gaze data and facial data are collected not just for recognition but for inferring mood, attention, and intent. Beyond the face, gate data (how a person walks) offers unique identifiers. Surprisingly, even spatial audio technology can contribute to identity, as every individual's ear structure processes sound uniquely. This data, when combined, allows for sophisticated inferences about a user's physical and mental state, potentially leading to discriminatory practices like denial of insurance coverage for pre-existing conditions.
  • Brain-Computer Interfaces (BCI): The future of data collection extends to direct neural input. Pearlman discusses both non-invasive and invasive BCIs, referencing Elon Musk's Neuralink project, which has demonstrated monkeys playing Pong via brain implants. The intersection of BCIs with XR will allow for an unprecedented fusion of digital experience and direct cognitive data, blurring the lines between thought and action.
  • Robotics Integration: A proof of concept already exists for "virtual tourism," where a user can control a robot remotely via an XR headset, allowing the robot to tour a location in 360 degrees on their behalf. This scenario highlights the potential for remote presence and interaction, further expanding the modalities of data capture and physical-digital convergence.

Novel Attack Vectors:

The shift in the attack surface from network infrastructure to the user's immediate environment and even their cognitive space gives rise to the four specific cyberattacks identified by the University of New Haven team:

  • Tracker Attack: This involves compromising a computer to which an XR device is tethered. Once compromised, the attacker can hijack the device's front-facing camera. This allows for real-time video streaming of the user's physical surroundings, effectively turning the XR headset into a surveillance device broadcasting the user's "living room" to the attacker.
  • Chaperone Attack: VR environments often utilize "chaperone" systems, which are virtual boundaries designed to prevent users from colliding with real-world objects. The attack involves manipulating these boundaries, either expanding them to trick a user into moving into an unsafe area or shrinking them to disorient or restrict movement.
  • Human Joystick Attack: This sophisticated attack leverages the manipulation of chaperone boundaries. By subtly and continuously altering these virtual perimeters, an attacker can coerce a user to move across a physical room without their conscious awareness or consent, treating the user as a "human joystick." This highlights the potential for physical manipulation and control within an immersive environment.
  • Ransomware Attack: In this scenario, an attacker gains control over the XR device's display. The entire visual output—what the user sees—is then replaced with a ransom demand or other malicious content, rendering the device unusable until the attacker's demands are met. This is a direct evolution of traditional ransomware into the immersive digital realm.

These technical details underscore Pearlman's urgent call for proactive measures, as the capabilities for data collection and manipulation in XR far exceed previous technologies, presenting risks that are both deeper and more pervasive.

Demo / Proof of Concept

▶ Watch: Specific cyber attacks: tracker, human joystick, ransomware (27:45)

Kavya Pearlman integrated several demonstrations and discussions of proofs of concept to illustrate both the promise and peril of XR technologies.

She first showcased a brief video of her co-founder, Marco Maniano, playing an augmented reality game at the Augmented World Expo (AWE). This demonstration depicted two individuals in a shared physical space, wearing AR glasses, interacting with digital "bugs" overlaid onto their real-world environment. The visual highlighted the collaborative and interactive potential of AR, where digital elements seamlessly integrate with the physical world, offering a glimpse into shared immersive experiences beyond mere passive observation.

To emphasize the enterprise and collaborative potential, Pearlman then presented a video from Qualcomm. This clip depicted individuals in an XR environment collaborating on a 3D model, specifically discussing and changing the "upholstery color" of "model 5a." This illustrated how XR facilitates design, prototyping, and remote collaboration, allowing geographically dispersed teams to interact with virtual objects as if they were physically present. While exciting, Pearlman quickly pivoted to the "terrifying" implications of such pervasive interaction and data capture.

Finally, she referenced a proof of concept involving robotics and XR for "virtual tourism." In this scenario, users could don a virtual reality headset and remotely control a robot equipped with 360-degree cameras to tour a physical location on their behalf. This concept, potentially accelerated by future global health crises, demonstrates how XR could enable immersive experiences and remote presence, further blurring the lines between physical and virtual existence and raising questions about the data collected from both the user and the remote environment.

Defensive Implications

▶ Watch: XR's 'two-way sword': healing vs. weaponizing mental health (35:30)

Recognizing the profound risks inherent in the era of constant reality capture, Kavya Pearlman outlines a robust defensive strategy, spearheaded by the XR Safety Initiative (XRSI). The core philosophy is to move beyond simply fighting for privacy to actively "protect the bits and bytes" of user data, ensure physical and psychological safety, and guard against device tampering.

Pearlman stresses that current regulations are insufficient and require a significant "upgrade and uplift" to address the unique challenges of XR. To this end, XRSI has formed broad coalitions with multi-disciplinary experts, including technologists, ethicists, legal scholars, and policymakers. Their collective effort aims to establish foundational security controls and safety controls. These controls are designed not only to protect technical vulnerabilities but also to mitigate pervasive social risks such as harassment, bullying, and the need for effective content moderation within immersive environments.

A tangible outcome of XRSI's work is the development of the XR Safety Privacy Framework. Informed by recent "Facebook disclosures" and whistleblowers, this framework provides a structured approach to identifying and addressing security and privacy risks. Pearlman indicates that version 2.0 of this framework is actively under development, signifying an ongoing commitment to evolving standards as the technology matures. XRSI is also actively engaged in advising "several global governments" on how to build informed policy around these emerging technologies.

Pearlman's call to action is clear: the responsibility for a safe XR future falls on everyone. Quoting Mother Teresa, she emphasizes that while individuals may have unique skills, "together we can do great things." This collaborative ethos is crucial for developing the necessary taxonomy, standards, and regulations that can inform policymakers and ensure that the benefits of XR are realized without compromising fundamental human rights and safety. The objective is to proactively shape the future of these technologies rather than reacting to crises, ensuring that identity protection, ethical data governance, and user well-being are at the forefront of development.

Key Takeaways

  • The Era of Constant Reality Capture: XR and the Metaverse are ushering in an unprecedented era of data collection, moving beyond traditional online data to capture highly intimate biometric and behavioral information, potentially yielding over 2 million data points on an individual in 20 minutes in VR.
  • Weaponization of Data: The vast amounts of personal and biometrically inferred data (gaze, facial, gait, spatial audio) collected in XR environments can be weaponized for targeted manipulation, surveillance, and discrimination, echoing and amplifying historical abuses seen with social media data.
  • Shifting Attack Surface: The traditional cybersecurity attack surface has expanded from networks and servers to encompass the user's physical living space and even their cognitive processes, leading to novel threats like tracker, chaperone, human joystick, and ransomware attacks specific to immersive environments.
  • Convergence Amplifies Risks: The integration of XR with advanced technologies such as AI, 5G/6G, robotics, and Brain-Computer Interfaces (BCI) will further intensify data collection, enabling deeper inferences about individuals and creating new vectors for exploitation if not properly secured.
  • Urgent Need for Standards and Regulation: There is a critical regulatory void regarding XR data. Proactive development of ethical frameworks, security controls, and safety standards—like those advocated by XRSI—is essential to prevent a "Collingwood Dilemma" where technology outpaces our ability to mitigate its harm.
  • Balancing Innovation with Protection: While XR offers transformative benefits in areas like healthcare, collaboration, and environmental sustainability, these must be balanced with robust identity protection, transparent data governance, and comprehensive user safety mechanisms to ensure a secure, inclusive, and trustworthy digital future.

About the Speaker(s)

Kavya Pearlman is a globally recognized expert and leading voice in the field of XR safety, privacy, and ethics. She is the founder and CEO of the XR Safety Initiative (XRSI), a non-profit organization dedicated to establishing standards and best practices for emerging immersive technologies. Pearlman's extensive background includes serving as the Head of Security for Linden Lab, the company behind the groundbreaking virtual world Second Life, and working in third-party security advising for Facebook in 2016, where she gained firsthand insight into the data and privacy challenges posed by large tech platforms. She is actively involved in advising global governments and organizations on policy and frameworks to ensure the safe and ethical development of the Metaverse and XR technologies.

Reviews

Dr. Zero (Offensive Security Researcher) — WEAK

A high-level policy primer that relies on Metaverse buzzwords and recycled statistics to pitch an awareness framework. Pearlman correctly identifies the privacy nightmare of XR telemetry, but offers zero technical depth or actionable mitigations for anyone actually building or defending these systems.

Heather Calloway (CISO) — STRONG ACCEPT

Pearlman delivers a necessary, unsentimental look at the incoming privacy and compliance nightmare of XR environments. The Metaverse hype may have cooled, but enterprise adoption of AR hardware hasn't, and this talk clearly defines the board-level risks of 'constant reality capture.' Required viewing for any leadership team considering deploying immersive hardware to their workforce.

→ Top-rated talks at Swiss Cyber Storm 2021

All talks from Swiss Cyber Storm 2021