Semantic Scene Completion from a Single Depth Image

Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva, Thomas Funkhouser
CVPR 2017 Oral

Semantic Scene Completion from a Single Depth Image

Abstract

This paper focuses on semantic scene completion, a task for producing a complete 3D voxel representation of volumetric occupancy and semantic labels for a scene from a single-view depth map observation. Previous work has considered scene completion and semantic labeling of depth maps separately. However, we observe that these two problems are tightly intertwined. To leverage the coupled nature of these two tasks, we introduce the semantic scene completion network (SSCNet), an end-to-end 3D convolutional network that takes a single depth image as input and simultaneously outputs occupancy and semantic labels for all voxels in the camera view frustum. Our network uses a dilation-based 3D context module to efficiently expand the receptive field and enable 3D context learning. To train our network, we construct SUNCG - a manually created large-scale dataset of synthetic 3D scenes with dense volumetric annotations. Our experiments demonstrate that the joint model outperforms methods addressing each task in isolation and outperforms alternative approaches on the semantic scene completion task.

Video

Paper

Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva, Thomas Funkhouser
Semantic Scene Completion from a Single Depth Image
CVPR 2017 Oral

Code

paper
github.com/shurans/sscnet

Citation

@article{song2016ssc,
  author     = {Song, Shuran and Yu, Fisher  and Zeng, Andy and Chang, Angel X and Savva, Manolis and Funkhouser, Thomas},
  title      = {Semantic Scene Completion from a Single Depth Image},
  journal 	 = {Proceedings of 30th IEEE Conference on Computer Vision and Pattern Recognition},
  year       = {2017},
}

Related


Interactive 3D Modeling with a Generative Adversarial Network

Interactive 3D Modeling with a Generative Adversarial Network

3DV 2017 We propose using a generative adversarial network (GAN) to assist a novice user in designing real-world shapes with a simple interface.


SHREC’17 Track Large-Scale 3D Shape Retrieval from ShapeNet Core55

SHREC’17 Track Large-Scale 3D Shape Retrieval from ShapeNet Core55

3DOR 2017 This track provides a benchmark to evaluate large-scale 3D shape retrieval based on the ShapeNet dataset.


ShapeNet: An Information-Rich 3D Model Repository

ShapeNet: An Information-Rich 3D Model Repository

arXiv 2015 We present ShapeNet: a richly-annotated, large-scale repository of shapes represented by 3D CAD models of objects.