A Survey of 3D Indoor Scene Synthesis

Indoor scene synthesis has become a popular topic in recent years. Synthesizing functional and plausible indoor scenes is an inherently difficult task since it requires considerable knowledge to both choose reasonable object categories and arrange objects appropriately. In this survey, we propose four criteria which group a wide range of 3D (three-dimensional) indoor scene synthesis techniques according to various aspects (specifically, four groups of categories). It also provides hints, through comprehensively comparing all the techniques to demonstrate their effectiveness and drawbacks, and discussions of potential remaining problems.

[1]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[2]  Yuan Liang,et al.  Automatic Data-Driven Room Design Generation , 2017, AniNex.

[3]  Antonio Torralba,et al.  Parsing IKEA Objects: Fine Pose Estimation , 2013, 2013 IEEE International Conference on Computer Vision.

[4]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[5]  Wilmot Li,et al.  Style compatibility for 3D furniture models , 2015, ACM Trans. Graph..

[6]  Christopher Potts,et al.  Text to 3D Scene Generation with Rich Lexical Grounding , 2015, ACL.

[7]  Ligang Liu,et al.  MIQP‐based Layout Design for Building Interiors , 2018, Comput. Graph. Forum.

[8]  Vladlen Koltun,et al.  Computer-generated residential building layouts , 2010, ACM Trans. Graph..

[9]  Sunghun Jo,et al.  GPU-Driven Scalable Parser for OBJ Models , 2018, Journal of Computer Science and Technology.

[10]  Duc Thanh Nguyen,et al.  SceneNN: A Scene Meshes Dataset with aNNotations , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[11]  Kai Wang,et al.  Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Ming Ouhyoung,et al.  On Visual Similarity Based 3D Model Retrieval , 2003, Comput. Graph. Forum.

[13]  Angel X. Chang,et al.  Interactive Learning of Spatial Knowledge for Text to 3D Scene Generation , 2014 .

[14]  Pat Hanrahan,et al.  Example-based synthesis of 3D object arrangements , 2012, ACM Trans. Graph..

[15]  Antonio Torralba,et al.  FPM: Fine Pose Parts-Based Model with 3D CAD Models , 2014, ECCV.

[16]  Lawrence Carin,et al.  A convergence analysis for a class of practical variance-reduction stochastic gradient MCMC , 2018, Science China Information Sciences.

[17]  Pat Hanrahan,et al.  Semantically-enriched 3D models for common-sense knowledge , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[18]  Jianzhong Qiao,et al.  Modeling the Correlations of Relations for Knowledge Graph Embedding , 2018, Journal of Computer Science and Technology.

[19]  Qinping Zhao,et al.  3D shape co-segmentation via sparse and low rank representations , 2017, Science China Information Sciences.

[20]  Matthias Nießner,et al.  PiGraphs , 2016, SIGGRAPH ASIA 2016 Virtual Reality meets Physical Reality: Modelling and Simulating Virtual Humans and Environments.

[21]  Daniel Cohen-Or,et al.  GRAINS , 2018, ACM Trans. Graph..

[22]  Pat Hanrahan,et al.  Synthesizing open worlds with constraints using locally annealed reversible jump MCMC , 2012, ACM Trans. Graph..

[23]  Tieniu Tan,et al.  DF2Net: Discriminative Feature Learning and Fusion Network for RGB-D Indoor Scene Classification , 2018, AAAI.

[24]  Andrew Owens,et al.  SUN3D: A Database of Big Spaces Reconstructed Using SfM and Object Labels , 2013, 2013 IEEE International Conference on Computer Vision.

[25]  Martial Hebert,et al.  Data-Driven Scene Understanding from 3D Models , 2012, BMVC.

[26]  N. Mitra,et al.  Exploration of continuous variability in collections of 3D shapes , 2011, SIGGRAPH 2011.

[27]  Olga Sorkine-Hornung,et al.  Object detection and classification from large‐scale cluttered indoor scans , 2014, Comput. Graph. Forum.

[28]  X. Jin.  Factor graphs and the Sum-Product Algorithm , 2002 .

[29]  Pat Hanrahan,et al.  On being the right scale: sizing large collections of 3D models , 2014, SIGGRAPH ASIA Indoor Scene Understanding Where Graphics Meets Vision.

[30]  Demetri Terzopoulos,et al.  The Clutterpalette: An Interactive Tool for Detailing Indoor Scenes , 2016, IEEE Transactions on Visualization and Computer Graphics.

[31]  Matthias Nießner,et al.  Activity-centric scene synthesis for functional 3D scene modeling , 2015, ACM Trans. Graph..

[32]  Avideh Zakhor,et al.  Planar 3D modeling of building interiors from point cloud data , 2012, 2012 19th IEEE International Conference on Image Processing.

[33]  Shi-Min Hu,et al.  3D indoor scene modeling from RGB-D data: a survey , 2015, Computational Visual Media.

[34]  Paul L. Rosin,et al.  Intelligent Visual Media Processing: When Graphics Meets Vision , 2017, Journal of Computer Science and Technology.

[35]  Stefan Leutenegger,et al.  SceneNet RGB-D: Can 5M Synthetic Images Beat Generic ImageNet Pre-training on Indoor Segmentation? , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[36]  Leonidas J. Guibas,et al.  Cross-Modal Attribute Transfer for Rescaling 3D Models , 2017, 2017 International Conference on 3D Vision (3DV).

[37]  Angel X. Chang,et al.  SceneSeer: 3D Scene Design with Natural Language , 2017, ArXiv.

[38]  Leonidas J. Guibas,et al.  Acquiring 3D indoor environments with variability and repetition , 2012, ACM Trans. Graph..

[39]  Chenfanfu Jiang,et al.  Human-Centric Indoor Scene Synthesis Using Stochastic Grammar , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[40]  Matthias Nießner,et al.  Scan2CAD: Learning CAD Model Alignment in RGB-D Scans , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[41]  Ramesh Nallapati,et al.  Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora , 2009, EMNLP.

[42]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[43]  E. Reed.  The Ecological Approach to Visual Perception , 1989 .

[44]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[45]  Maneesh Agrawala,et al.  Interactive furniture layout using interior design guidelines , 2011, SIGGRAPH 2011.

[46]  Matthias Nießner,et al.  ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Silvio Savarese,et al.  Toward coherent object detection and scene layout understanding , 2011, Image Vis. Comput..

[48]  Yuandong Tian,et al.  Semantic Amodal Segmentation , 2015, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[49]  Rui Ma,et al.  Organizing heterogeneous scene collections through contextual focal points , 2014, ACM Trans. Graph..

[50]  T. Germer,et al.  Procedural Arrangement of Furniture for Real‐Time Walkthroughs , 2009, Comput. Graph. Forum.

[51]  H. Zhang,et al.  Learning 3D Scene Synthesis from Annotated RGB‐D Images , 2016, Comput. Graph. Forum.

[52]  Roberto Cipolla,et al.  SceneNet: Understanding Real World Indoor Scenes With Synthetic Data , 2015, ArXiv.

[53]  Zhongke Wu,et al.  Isometric 3D Shape Partial Matching Using GD-DNA , 2018, Journal of Computer Science and Technology.

[54]  Leonidas J. Guibas,et al.  ShapeNet: An Information-Rich 3D Model Repository , 2015, ArXiv.

[55]  Thomas A. Funkhouser,et al.  Semantic Scene Completion from a Single Depth Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Yun Jiang,et al.  Learning Object Arrangements in 3D Scenes using Human Context , 2012, ICML.

[57]  Chi-Keung Tang,et al.  Make it home: automatic optimization of furniture arrangement , 2011, ACM Trans. Graph..

[58]  Yujie Liu,et al.  MagicMark: a marking menu using 2D direction and 3D depth information , 2018, Science China Information Sciences.

[59]  Angel X. Chang,et al.  Learning Spatial Knowledge for Text to 3D Scene Generation , 2014, EMNLP.

[60]  Hailong Sun,et al.  Collusion-Proof Result Inference in Crowdsourcing , 2018, Journal of Computer Science and Technology.

[61]  Maneesh Agrawala,et al.  SceneSuggest: Context-driven 3D Scene Design , 2017, ArXiv.

[62]  Nathan Silberman,et al.  Indoor scene segmentation using a structured light sensor , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[63]  Chi-Keung Tang,et al.  Make it home: automatic optimization of furniture arrangement , 2011, SIGGRAPH 2011.

[64]  Thorsten Joachims,et al.  Contextually guided semantic labeling and search for three-dimensional point clouds , 2013, Int. J. Robotics Res..

[65]  Pat Hanrahan,et al.  Characterizing structural relationships in scenes using graph kernels , 2011, SIGGRAPH 2011.

[66]  Claude Berge,et al.  Hypergraphs - combinatorics of finite sets , 1989, North-Holland mathematical library.

[67]  Leonidas J. Guibas,et al.  Language-driven synthesis of 3D scenes from scene databases , 2018, ACM Trans. Graph..

[68]  Rui Ma,et al.  Action-driven 3D indoor scene evolution , 2016, ACM Trans. Graph..

[69]  Yun Jiang,et al.  Modeling 3D Environments through Hidden Human Context , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[70]  Yun Jiang,et al.  Learning to place new objects in a scene , 2012, Int. J. Robotics Res..

[71]  Ke Xie,et al.  A search-classify approach for cluttered indoor scene understanding , 2012, ACM Trans. Graph..

[72]  Hang Yang,et al.  Structured Indoor Modeling , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[73]  Pat Hanrahan,et al.  SceneGrok: inferring action maps in 3D environments , 2014, ACM Trans. Graph..

[74]  Shi-Min Hu,et al.  Sketch2Scene: sketch-based co-retrieval and co-placement of 3D models , 2013, ACM Trans. Graph..

[75]  Jianxiong Xiao,et al.  SUN RGB-D: A RGB-D scene understanding benchmark suite , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[76]  Dieter Fox,et al.  Unsupervised feature learning for 3D scene labeling , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[77]  Jie Xu,et al.  View suggestion for interactive segmentation of indoor scenes , 2017, Computational Visual Media.

[78]  Bin Zhou,et al.  Adaptive synthesis of indoor scenes via activity-associated object relation graphs , 2017, ACM Trans. Graph..

[79]  Nir Friedman,et al.  Bayesian Network Classifiers , 1997, Machine Learning.

[80]  Wei Wang,et al.  Effectively modeling piecewise planar urban scenes based on structure priors and CNN , 2018, Science China Information Sciences.

[81]  Shi-Min Hu,et al.  Structure recovery by part assembly , 2012, ACM Trans. Graph..

[82]  Fei Xu,et al.  Knowledge graph construction with structure and parameter learning for indoor scene design , 2018, Computational Visual Media.

[83]  I ScottKirkpatrick.  Optimization by Simulated Annealing: Quantitative Studies , 1984 .

[84]  Pat Hanrahan,et al.  Context-based search for 3D models , 2010, ACM Trans. Graph..

[85]  Kun Zhou,et al.  An interactive approach to semantic modeling of indoor scenes with an RGBD camera , 2012, ACM Trans. Graph..

[86]  Derek Hoiem,et al.  Indoor Segmentation and Support Inference from RGBD Images , 2012, ECCV.

[87]  Angel X. Chang,et al.  Deep convolutional priors for indoor scene synthesis , 2018, ACM Trans. Graph..

[88]  Bin Wang,et al.  Wall grid structure for interior scene synthesis , 2015, Comput. Graph..

[89]  Marc Alexa,et al.  Sketch-based shape retrieval , 2012, ACM Trans. Graph..

[90]  Radomír Mech,et al.  Metropolis procedural modeling , 2011, TOGS.

[91]  Kang Chen,et al.  Automatic semantic modeling of indoor scenes from low-quality RGB-D data using contextual information , 2014, ACM Trans. Graph..

[92]  Wenbin Li,et al.  InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset , 2018, BMVC.

[93]  Li Fei-Fei,et al.  Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image Retrieval , 2015, VL@EMNLP.

[94]  Roberto Cipolla,et al.  SceneNet: An annotated model generator for indoor scene understanding , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[95]  Guoliang Li,et al.  Crowdsourced Data Management: Overview and Challenges , 2017, SIGMOD Conference.

[96]  Bin Wang,et al.  Reshuffle-based interior scene synthesis , 2013, VRCAI '13.

[97]  Andrew Y. Ng,et al.  Parsing Natural Scenes and Natural Language with Recursive Neural Networks , 2011, ICML.