ClusterSplatSemantic Cluster Selection for 3D Visual Grounding in Gaussian Splatting

Jaesung Lee, Hyeontaek Hwang, Gabriel Manalu, Daeyoung Kim

School of Computing, KAIST

NeurIPS 2026 Spotlight

Ground a referring expression by selecting a coherent 3D cluster, with rendered 2D masks and an explicit set of target Gaussians.

A bed with blue sheets is described relative to a white trash can. Primitive-level top-ratio filtering scatters the target, while scene-adaptive cluster selection isolates the bed coherently.
From score filtering to target selection. Instead of extracting a 3D object from unstable primitive-level relevance scores, ClusterSplat selects a scene-adaptive cluster conditioned on the referring expression.

The idea

Language grounding with explicit 3D targets

3D visual grounding asks a referring expression to identify an object both in rendered views and in the 3D scene. Dense Gaussian relevance scores can change across scenes and prompts, making threshold or top-ratio selection sensitive to the chosen heuristic.

ClusterSplat reframes this task as selection over scene-adaptive candidates. It learns instance-aware Gaussian features, forms feature-aware seed voxels, and merges neighboring seeds when a description-length criterion improves. A query-conditioned scorer ranks the resulting clusters: the Top-1 cluster defines the explicit 3D target, while Gaussian-assigned cluster scores support differentiable 2D mask rendering.

Method

Prepare the scene. Select the target.

Scene structure is learned once; each referring expression conditions the cluster scorer.

The four-stage pipeline learns instance-aware features from multi-view masks, builds scene-adaptive clusters, scores clusters using CLIP text tokens and attention, and produces an explicit 3D target together with a rendered 2D mask.
Instance-aware feature learning and description-length merging create candidates. Query-conditioned scoring selects one 3D cluster; cluster logits assigned to the Gaussians are rendered to predict the 2D mask.
  1. 01

    Learn instance-aware features

    Multi-view mask supervision encourages Gaussians from the same instance to share features while separating different instances.

  2. 02

    Build scene-adaptive candidates

    Feature-aware seed voxels merge under a description-length criterion. Each scene determines its own number of candidate clusters.

  3. 03

    Ground the expression

    A text-conditioned scorer ranks the candidates. Top-1 selection defines the 3D target, and score rendering predicts the 2D mask.

Interactive examples

Explore the grounding results

Choose a description and inspect the target. Watch the target along a camera path, or compare matched results in Rendered 2D and 3D selection.

Choose from the interactive examples when they have loaded, or view the complete figures below.

Methods in motion

Watch the scene and selected target along the camera path. Choose Rendered 2D or 3D selection to compare methods.

Qualitative results

Rendered masks and 3D object selection

Comparisons from the paper, using appearance and spatial descriptions to identify the target.

ScanRefer

Each example pairs a rendered result with a top-down 3D view, showing both image-space localization and explicit target selection.

Four ScanRefer examples compare ground truth, LangSplat, OpenGaussian, Dr.Splat, ReferSplat, and ClusterSplat for a trash can, a picture, a toilet, and a desk. Each row includes rendered masks and top-down 3D target selection.
ScanRefer descriptions combine object appearance and spatial relations. In the 3D visualizations, selected target Gaussians are yellow and background Gaussians are blue. Open full-resolution figure ↗

Quantitative results

Across referring and class-name queries

Results reported in the manuscript. All values use a 0–100 scale; higher is better.

ScanRefer ScanNet v2

ScanRefer

ScanRefer: referring-expression grounding across 10 scenes.
Method2D mIoU ↑2D mAcc@0.25 ↑3D mIoU ↑3D mAcc@0.25 ↑
LangSplat11.0013.415.814.95
OpenGaussian18.485.531.060.10
Dr.Splat19.0127.769.4811.17
ReferSplat31.3045.074.050.10
ReferSplat w/ our loss31.5545.993.850.00
ClusterSplat39.7254.3523.3343.38

ScanNet v2

ScanNet v2: class-name queries across the same 10 scenes.
Method2D mIoU ↑2D mAcc@0.25 ↑3D mIoU ↑3D mAcc ↑
LangSplat27.6740.822.018.13
OpenGaussian19.5526.1131.0546.39
Dr.Splat19.9432.6430.6746.12
ReferSplat28.7040.126.3517.22
ReferSplat w/ our loss32.7047.324.5212.89
ClusterSplat42.2058.3029.2146.60

Bold values mark the best score in each column. ScanRefer and ScanNet v2 use a common set of 10 scenes. Score-based baselines use the best per-method top-ratio within 0.01–0.10 under the manuscript's oracle sweep; ClusterSplat selects the Top-1 cluster.

ClusterSplat has the highest ScanRefer scores among the compared methods. On ScanNet v2 it leads the rendered 2D metrics and 3D matching accuracy; OpenGaussian has the higher 3D mIoU.

Scope

Candidate quality matters

ClusterSplat selects one candidate for each referring expression. Over-merging can include nearby distractors, while fragmentation can leave part of an object outside the selected cluster. The quality of scene-adaptive candidates therefore limits the final target selection.