Beyond Isolated Objects: Relationship-aware Open-Vocabulary 3D Scene Understanding via 3D Scene Graph Analysis

ECCV 2026

Xianhao Chen1* Jiarui Hu1* Yuanbo Yang2 Xiyu Zhang1 Tengyue Wang2 Hujun Bao1 Guofeng Zhang1 Zhaopeng Cui1†
1State Key Lab of CAD&CG, Zhejiang University 2Zhejiang University
*Equal contribution. †Corresponding author.
Paper arXiv Code Video
RelGraphOV teaser
TL;DR RelGraphOV turns a scene into a relationship-aware 3D scene graph and refines open-vocabulary features with a dual-stream graph network, so the surrounding context—not appearance alone—decides each object's label (e.g., curtain vs. shower curtain). It achieves state-of-the-art open-vocabulary 3D segmentation on ScanNet, ScanNet200, ScanNet++, and Replica.

Abstract

Open-vocabulary 3D scene understanding aims to segment 3D scenes beyond predefined categories by transferring semantic knowledge from vision-language models. Existing methods have advanced this task by lifting language-aligned 2D features into 3D, yet they often rely on context-independent semantic representations, leaving object relationships underexplored for contextual refinement. We propose RelGraphOV, a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding. Our method constructs relational scene graphs from multi-view observations by leveraging vision-language reasoning to infer object relationships and prune geometrically implausible connections, without manual relationship annotations. To aggregate relational context while avoiding feature interference, we introduce an Adaptive Gated Dual-Stream Contextual GAT that separates dense geometric features and semantic CLIP embeddings, performs edge-guided message passing, and adaptively fuses complementary semantics. A hierarchical contrastive objective further promotes instance-level consistency and category-level discrimination. Experiments on ScanNet, ScanNet200, ScanNet++, and Replica demonstrate strong performance and generalization ability.

Keywords: 3D Semantic Segmentation · Scene Graph · Open-Vocabulary

Video

Method

RelGraphOV pipeline

Given posed RGB-D observations, RelGraphOV refines open-vocabulary 3D semantics in four stages:

  1. Relationship-aware Scene Graph Abstract objects into graph nodes and reason about their relations with a vision-language model, pruning geometrically implausible edges.
  2. Multi-view Node Annotation A VLM data engine generates rich node descriptions with cross-view consistency adjudication, providing supervision without manual labels.
  3. Adaptive Gated Dual-Stream GAT Decouple dense geometric and CLIP semantic streams and fuse them through a learnable gate during edge-guided message passing, avoiding feature interference.
  4. Hierarchical Contrastive Training A hierarchical contrastive objective with dual-alignment losses mitigates semantic drift and catastrophic forgetting.

Experiments

Qualitative Results

BibTeX

@inproceedings{chen2026relgraphov,
  title     = {Beyond Isolated Objects: Relationship-aware Open-Vocabulary
               3D Scene Understanding via 3D Scene Graph Analysis},
  author    = {Chen, Xianhao and Hu, Jiarui and Yang, Yuanbo and Zhang, Xiyu and
               Wang, Tengyue and Bao, Hujun and Zhang, Guofeng and Cui, Zhaopeng},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2026}
}