ReferSplat: Referring Segmentation in 3D Gaussian Splatting

ReferSplat: Referring Segmentation in 3D Gaussian Splatting

Speakers: Shuting He, Guangquan Jie, Changshuo Wang, Yun Zhou, Shuming Hu, Guanbin Li, Henghui Ding (Presented by Tang Xiaoli, Nanyang Technological University)

Conference: ICML 2025

YouTube: https://slideslive.com/39043890

Overview

The talk "ReferSplat: Referring Segmentation in 3D Gaussian Splatting" introduces a groundbreaking framework for enabling natural language interaction with 3D scenes, specifically within the context of 3D Gaussian Splatting (3DGS). Presented by Tang Xiaoli from Nanyang Technological University on behalf of a collaborative team from Shanghai University of Finance and Economics, Fudan University, Nanyang Technological University, and Sun Yat-sen University, this spotlight paper addresses a critical gap in 3D scene understanding: the ability to interpret and localize objects based on arbitrary, free-form natural language descriptions. Traditional methods often rely on fixed-pattern class names, severely limiting their applicability in dynamic, real-world scenarios.

To bridge this limitation, the authors propose a novel task called Referring 3D Gaussian Splatting Segmentation (R3DGS), which aims to segment objects in a 3D Gaussian scene using natural language expressions that typically encode complex spatial relationships and descriptive attributes. The core contribution, ReferSplat, is an end-to-end framework designed to model 3D Gaussian points with language expressions in a spatially-aware paradigm. This research is highly significant for advancing immersive technologies like VR/AR systems, robotics, and intelligent scene manipulation, where intuitive human-computer interaction with 3D environments is paramount.

Background

▶ Watch: Introduction to ReferSplat and R3DGS task (0:00)

The landscape of neural rendering has been significantly reshaped by 3D Gaussian Splatting (3DGS), a recently proposed technique lauded for its rapid training times, real-time rendering capabilities, and explicit point-based representation. These attributes have propelled 3DGS to the forefront of 3D scene reconstruction and visualization. Concurrently, the field of text-driven 3D scene understanding has gained considerable traction, particularly in areas like Open-Vocabulary 3DGS Segmentation (3DOVS). While 3DOVS allows for the segmentation of objects based on linguistic class names, it is fundamentally constrained by its reliance on fixed, predefined patterns. This limitation means users can only query objects using specific labels, rather than descriptive, context-rich language.

The inherent problem lies in the inability of existing systems to handle free-form natural language interactions with 3D scenes. Real-world applications, from advanced VR/AR experiences to sophisticated robotic navigation, demand a more nuanced understanding of user intent, often expressed through complex sentences that describe spatial relationships (e.g., "the cup placed near the keyboard") or fine-grained attributes (e.g., "the round, red ball on the surface"). Current methods fall short here, as they struggle to interpret these arbitrary descriptions and localize the corresponding objects in 3D space, especially when objects are occluded or not directly visible from a given viewpoint.

This unmet need motivated the introduction of Referring 3D Gaussian Splatting Segmentation (R3DGS). R3DGS represents a paradigm shift, focusing on segmenting objects in a 3D Gaussian scene based on natural language expressions that are rich in spatial relationships and descriptive attributes. For instance, an R3DGS model should be able to identify "the blue book on the top shelf" even if parts of it are obscured. A straightforward approach might be to adapt existing open-vocabulary 3D scene understanding methods by feeding them complex natural language expressions instead of simple class names. However, these methods typically project semantic features onto 2D images for pixel-level understanding, leveraging semantic knowledge from pre-trained 2D vision-language models to guide 3D scene representation learning. During inference, output masks are generated by matching text queries with rendered 2D features.

This conventional approach suffers from several critical drawbacks when applied to R3DGS. Firstly, there is a fundamental lack of direct interaction between the text query and the 3D Gaussian representations during training. The system relies on matching 2D rendered features, rather than performing localization directly within the 3D space, which severely limits performance in complex, occluded scenarios. Secondly, this training paradigm inherently overlooks crucial position information. Rendered 2D features cannot intrinsically understand the intricate spatial relationships between objects described in a sentence. Instead, they primarily focus on semantic identification, rendering them ineffective for tasks that demand sophisticated spatial reasoning. This highlights a clear need for a new network architecture capable of modeling 3D Gaussians with language expressions in a truly spatially-aware manner.

Key Findings

▶ Watch: Limitations of existing open-vocabulary 3D methods (4:00)

The research introduces ReferSplat, an innovative end-to-end framework designed to address the challenges of referring segmentation in 3D Gaussian Splatting. The core contributions and findings include:

• Novel Task and Dataset: The authors formally define Referring 3D Gaussian Splatting Segmentation (R3DGS), a new task requiring models to segment objects in 3D Gaussian scenes based on free-form natural language expressions incorporating spatial and descriptive attributes. To support this, they construct Ref-NeRF, a pioneering dataset characterized by its rich, complex, and spatially grounded language expressions, making it a robust benchmark for R3DGS.
• Spatially-Aware Framework: ReferSplat is presented as an end-to-end framework that models 3D Gaussian points with natural language expressions using a unique spatially-aware paradigm. This overcomes the limitations of prior 2D-centric methods by enabling direct interaction between text queries and 3D Gaussian representations.
• Referring Feature Fields: A critical innovation is the assignment of a referring feature vector to each 3D Gaussian. These vectors collectively form referring fields that interact directly with text queries during training, allowing for 3D scene knowledge to be leveraged for identifying occluded or non-visible objects.
• Position-Aware Cross-Modal Interaction (PCMI): To enhance spatial reasoning, ReferSplat incorporates a dedicated PCMI module. This module extracts and refines position features for both Gaussians and language descriptions through position-guided attention, effectively aligning 3D Gaussian representations with the nuances of textual spatial cues.
• Gaussian-Text Contrastive Learning (GTCL): To improve discriminative power and handle semantically similar expressions that refer to different objects, the framework employs GTCL. This technique performs contrastive learning between positive Gaussian embeddings (derived from selectively chosen positive Gaussian referring features) and text embeddings, thereby refining cross-modal understanding.
• State-of-the-Art Performance: ReferSplat significantly outperforms existing state-of-the-art 2D-based methods (like Grounded-SAM) and 3D-based approaches (such as LangSplat and Gaussian Grouping) on the challenging Ref-NeRF dataset. This demonstrates its superior effectiveness in referring 3DGS segmentation.
• Component Efficacy Validation: Comprehensive ablation studies rigorously validate the effectiveness of each proposed component. The referring feature fields, PCMI, and GTCL individually contribute substantial improvements to the model's performance, particularly in mIoU (mean Intersection over Union), confirming their necessity for accurate R3DGS.

Technical Deep Dive

▶ Watch: Enhancing spatial reasoning with PCMI module (6:00)

ReferSplat is meticulously designed as an end-to-end framework that integrates language perception directly into the 3D Gaussian Splatting representation, operating within a spatially-aware paradigm. The architecture comprises several innovative components that collectively enable robust referring 3DGS segmentation.

At its core, ReferSplat addresses the challenge of associating arbitrary natural language expressions with specific 3D Gaussians. To achieve this language perception, the framework assigns a unique referring feature vector to each individual 3D Gaussian. These vectors are not static; they are learned and dynamically updated, forming what the authors term "referring fields" across the entire 3D scene. During the training phase, these 3D Gaussian referring features actively interact with the textual queries. This interaction is facilitated by computing a similarity score between the textual features (derived from the input natural language expression) and the 3D Gaussian referring features. This similarity metric is crucial for localizing the described object. Subsequently, a 2D segmentation mask is obtained by projecting these similarity scores onto the rendered image plane, effectively highlighting the region corresponding to the textual description.

A key advantage of these constructed 3D Gaussian referring fields is their inherent ability to leverage 3D scene knowledge learned from multi-view training images. This enables the model to identify objects that are occluded or not directly visible from a single viewpoint—a significant limitation for purely 2D-based methods. For segmentation supervision, ReferSplat employs a clever strategy: it generates pseudo ground truth masks using a confidence-weighted IoU strategy. This approach provides robust training signals without requiring extensive manual 3D annotations for every referring expression.

To specifically enhance spatial reasoning, a critical component for interpreting descriptions like "the object behind the chair" or "the item on top of the table," ReferSplat introduces the Position-Aware Cross-Modal Interaction (PCMI) module. This module is designed to meticulously extract and refine positional information from both the 3D Gaussians and the language descriptions. For Gaussians, this might involve their 3D coordinates and local contextual features. For language, it involves parsing and understanding spatial prepositions and relative clauses. These extracted features are then refined through a sophisticated position-guided attention mechanism. This attention mechanism ensures that the 3D Gaussian representations are accurately aligned with the spatial nuances embedded within the text descriptions, allowing the model to grasp complex spatial relationships with high precision.

Despite the advancements in spatial awareness, sentences with similar semantics but referring to different target objects can still cause confusion, degrading performance. For example, "the red book" and "the blue book" might be semantically similar if both are books, but they refer to distinct objects. To address this fine-grained differentiation, ReferSplat incorporates Gaussian-Text Contrastive Learning (GTCL). This module is designed to construct more discriminative multi-modal representations. GTCL operates by performing contrastive learning between positive Gaussian embeddings and text embeddings. The positive Gaussian embeddings are computed from selectively chosen positive Gaussian referring features—those Gaussians that definitively belong to the target object described by the text. By contrasting these positive pairs against negative pairs (Gaussians not belonging to the target object, or text referring to other objects), the model is compelled to learn highly discriminative features. This process significantly improves the model's ability to differentiate between subtle variations in referring expressions, thereby enhancing overall cross-modal understanding and segmentation accuracy.

In summary, ReferSplat's technical prowess stems from its holistic integration of 3D Gaussian features with language understanding. The referring feature fields anchor language to 3D points, the PCMI module grounds spatial reasoning, and GTCL refines the discriminative power, making it a robust solution for the R3DGS task.

Experimental Setup & Results

▶ Watch: Improving cross-modal understanding with GTCL (6:40)

To rigorously evaluate ReferSplat, the authors introduced and utilized the Ref-NeRF dataset, a cornerstone of their experimental setup. This dataset was specifically constructed to support the newly defined Referring 3D Gaussian Splatting Segmentation (R3DGS) task, emphasizing complex and spatially grounded language expressions. A key characteristic of Ref-NeRF is its linguistic richness: the average sentence length exceeds 13.6 words, making it approximately eight times longer than sentences found in previous datasets like NeRF-OVS. The dataset's word cloud highlights a strong emphasis on spatial reasoning and detailed object understanding, featuring a significant portion of relative position words (e.g., "placed near," "next") and fine-grained object attributes (e.g., "round," "surface"). This complexity ensures that models trained on Ref-NeRF must develop a deep understanding of both language and spatial relationships, establishing it as a more realistic and comprehensive benchmark.

ReferSplat was evaluated against a spectrum of state-of-the-art methods, encompassing both 2D-based and 3D-based approaches.

• 2D-based baselines included Grounded-SAM, a prominent model known for generating high-quality segmentation masks.
• 3D-based baselines included LangSplat and Gaussian Grouping, which represent current advancements in 3D scene understanding.
• Additionally, the proposed method was compared against three adapted versions of existing open-vocabulary methods, modified to handle complex natural language expressions.

The primary evaluation metric used was mIoU (mean Intersection over Union), a standard measure for segmentation accuracy.

The results unequivocally demonstrated ReferSplat's superior performance:

• Overall Performance: ReferSplat significantly outperformed all evaluated 2D-based and 3D-based methods on the Ref-NeRF dataset. This confirms its effectiveness in handling the complexities of R3DGS. The inherent advantage of ReferSplat's 3D Gaussian referring fields allowed it to recognize occluded or non-visible objects by leveraging multi-view 3D scene knowledge—a capability fundamentally limited in 2D-based methods like Grounded-SAM, which are restricted to visible objects within a single view.

• Ablation Studies: Comprehensive ablation studies were conducted to dissect the contributions of each core component of ReferSplat:
• Position-Aware Cross-Modal Interaction (PCMI) and Referring Feature Fields: The incorporation of PCMI, along with the constructed referring feature fields (referred to as index one in the presentation's table), led to substantial improvements. It boosted mIoU by 5.1% and 4.3% respectively compared to the baseline configuration. This demonstrates that PCMI is crucial for enhancing spatial reasoning and improving feature alignment between 3D Gaussians and text.
• Gaussian-Text Contrastive Learning (GTCL): The addition of GTCL (index two in the table) further improved the model's ability to distinguish semantically similar expressions, leading to an additional mIoU improvement of 4.4% and 3.4% respectively. This highlights GTCL's role in constructing discriminative multi-modal representations.
• Comparison with Adapted Baselines: ReferSplat clearly outperformed all adapted versions of existing methods. This result validates that the proposed 3D referring feature fields are highly effective in capturing the intricate relationship between 3D Gaussians and language, superior to simply modifying existing paradigms.

• Pseudo Label Validation: To validate the quality of the pseudo ground truth masks generated by the confidence-weighted IoU strategy, the authors manually annotated a subset of ground truth masks. They found that their method achieved approximately 50% mIoU when compared against these true labels, significantly outperforming alternative pseudo-labeling strategies. This confirms the effectiveness and reliability of their pseudo-label generation approach.

• Cross-Modal Interaction Design: Further experiments confirmed that removing or simplifying the carefully designed cross-modal interaction components (including both position-aware elements) consistently led to performance drops, underscoring their essential role in accurate R3DGS.

• Performance on 3DOVS: Although not specifically tailored for Open-Vocabulary 3DGS Segmentation (3DOVS), ReferSplat achieved state-of-the-art results in this related task. This collateral benefit is attributed to the combined strength of its Gaussian referring fields, position-aware interaction, and contrastive learning, which collectively enhance spatial reasoning and language understanding across different 3D segmentation contexts.

These results collectively establish ReferSplat as a powerful and effective solution for referring segmentation in 3D Gaussian Splatting, setting a new benchmark for natural language interaction with 3D scenes.

Practical Implications

▶ Watch: Ablation studies validating PCMI and GTCL (7:50)

The introduction of ReferSplat and the R3DGS task carries significant practical implications for a wide range of stakeholders involved in AI/ML development and deployment, particularly those working with 3D environments.

For practitioners and infrastructure teams, ReferSplat ushers in an era of more intuitive and natural human-computer interaction with 3D scenes. By moving beyond the limitations of fixed-pattern linguistic class names (as seen in 3DOVS), it enables users to interact with 3D models and environments using free-form natural language expressions. This capability is transformative for applications such as:

• Virtual Reality (VR) and Augmented Reality (AR) systems: Users can now verbally describe objects in a virtual or augmented scene, such as "the blue mug on the table to my left," and have the system accurately identify and highlight it. This greatly enhances user experience and reduces cognitive load.
• Robotics: Robots operating in complex environments can receive more natural, descriptive commands (e.g., "pick up the small red box near the door") rather than needing precise coordinates or pre-labeled object IDs. This improves autonomy and adaptability.
• Digital Twins and Industrial Applications: Engineers or operators can query complex industrial digital twins using natural language to locate specific components, inspect their status, or identify parts for maintenance, streamlining operations and reducing errors.
• 3D Content Creation and Editing: Designers could use voice commands to select and manipulate specific elements within a 3D model, making the design process more fluid and accessible.

For model builders and deployers, ReferSplat provides a robust, end-to-end framework for tackling the challenging R3DGS task. The availability of the Ref-NeRF dataset is a crucial contribution, offering a high-quality benchmark rich in complex, spatially-grounded language expressions. This dataset allows researchers and developers to train and evaluate models against realistic user queries, fostering further innovation in the field. The architectural components—referring feature fields, position-aware cross-modal interaction, and Gaussian-text contrastive learning—offer modular insights into building effective language-grounded 3D systems. Deployers can leverage these insights to build systems that are more resilient to occlusions and capable of understanding nuanced spatial relationships, leading to more reliable and user-friendly deployments.

However, like any cutting-edge technology, ReferSplat comes with its tradeoffs and limitations. The primary challenge identified by the authors lies in scenarios involving significant viewpoint and perspective shifts. While ReferSplat demonstrates strong performance in recognizing occluded objects, extreme changes in viewpoint can still degrade its accuracy. This suggests that while the 3D Gaussian referring fields provide a degree of viewpoint invariance, there are still complexities in maintaining consistent language-to-3D grounding across highly diverse perspectives. For practitioners, this means that while ReferSplat excels in many common use cases, applications requiring extreme robustness to arbitrary viewpoints (e.g., highly dynamic drone navigation or highly variable user head poses in AR) might still require further research and development to achieve peak performance.

Future work aims to address these limitations by exploring more robust multi-modal representations and enhancing global scene understanding. This could involve incorporating more sophisticated attention mechanisms that aggregate information across wider scene contexts or developing adaptive strategies to handle viewpoint changes. For those looking to deploy ReferSplat-like systems, understanding these current limitations is crucial for setting realistic expectations and planning for future enhancements. Nonetheless, ReferSplat marks a significant leap forward in making 3D Gaussian Splatting truly interactive and intelligent through the power of natural language.

Key Takeaways

• New Task & Dataset: The paper introduces Referring 3D Gaussian Splatting Segmentation (R3DGS), enabling free-form natural language interaction with 3DGS, and releases the Ref-NeRF dataset, a complex benchmark with average sentence lengths exceeding 13.6 words.
• ReferSplat Framework: ReferSplat is an end-to-end, spatially-aware framework that models 3D Gaussians with language expressions for R3DGS, moving beyond fixed-pattern class names.
• Core Components: Key to ReferSplat are 3D Gaussian referring feature fields (for language perception and occluded object recognition), a Position-Aware Cross-Modal Interaction (PCMI) module (for spatial reasoning via position-guided attention), and Gaussian-Text Contrastive Learning (GTCL) (for differentiating fine-grained expressions).
• Superior Performance: ReferSplat significantly outperforms state-of-the-art 2D-based (e.g., Grounded-SAM) and 3D-based (e.g., LangSplat) methods on the Ref-NeRF dataset, demonstrating its effectiveness in complex 3D referring segmentation.
• Component Validation: Ablation studies confirm the critical role of each component, with PCMI improving mIoU by 5.1%/4.3% and GTCL by 4.4%/3.4%, validating their necessity for accurate R3DGS.
• Practical Impact & Limitations: ReferSplat enables more intuitive human-computer interaction for VR/AR and robotics but faces challenges in scenarios involving significant viewpoint and perspective shifts, indicating an area for future research into more robust global scene understanding.

About the Speaker(s)

The talk was presented by Tang Xiaoli from Nanyang Technological University. The research behind ReferSplat is a collaborative effort involving multiple institutions, including Shanghai University of Finance and Economics, Fudan University, Nanyang Technological University, and Sun Yat-sen University. The authors listed on the paper are Shuting He, Guangquan Jie, Changshuo Wang, Yun Zhou, Shuming Hu, Guanbin Li, and Henghui Ding, reflecting a broad academic partnership contributing to this advanced work in 3D scene understanding and neural rendering.