Webvr Human-Centered Indoor Layout Design Framework Using a Convolutional Neural Network and Deep Q-Learning

With the rapid development of Web Virtual Reality (WebVR) technology, increasing focus has been placed on this domain. WebVR indoor scenario design studies have been of important value in both academia and industry. However, many bottlenecks still need to be overcome, such as the weak computing capacity and limited memory space available in web browsers. In particular, there are many deficiencies in virtual scenes, i.e., lack of fidelity, low automation and poor scene interactability. In this paper, we propose a novel WebVR indoor furniture layout design framework to enhance the capabilities of automatic furniture layout and significantly enhance the interactiveness of virtual scenarios. In particular, we present a hand-drawn sketch recognition scheme based on the ResNet convolutional neural network (CNN), which can strongly improve scenario interactiveness by allowing the user to conveniently add new furniture by means of free-hand drawing operations rather than tedious manual drag-and-pull operations. In addition, based on a deep Q-learning network (DQN), the best positions (states) for these pieces of furniture (agents) in virtual scenarios can be automatically determined, making it easy to satisfy popular design principles. Finally, we report experiments conducted to validate the feasibility of our proposed framework, and the results fully demonstrate that this framework is completely feasible.

[1]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[2]  Daniel Cohen-Or,et al.  GRAINS , 2018, ACM Trans. Graph..

[3]  Thomas A. Funkhouser,et al.  Semantic Scene Completion from a Single Depth Image , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  C. Lawrence Zitnick,et al.  Bringing Semantics into Focus Using Visual Abstraction , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[5]  Eugene Zhang,et al.  Interactive procedural street modeling , 2008, ACM Trans. Graph..

[6]  Marc Alexa,et al.  How do humans sketch objects? , 2012, ACM Trans. Graph..

[7]  Tinne Tuytelaars,et al.  Sketch classification and classification-driven analysis using Fisher vectors , 2014, ACM Trans. Graph..

[8]  Levent Burak Kara,et al.  Sketch-Based 3D-Shape Creation for Industrial Styling Design , 2007, IEEE Computer Graphics and Applications.

[9]  Fang Wang,et al.  Sketch-based 3D shape retrieval using Convolutional Neural Networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Vladlen Koltun,et al.  Computer-generated residential building layouts , 2010, ACM Trans. Graph..

[11]  Chi-Keung Tang,et al.  Make it home: automatic optimization of furniture arrangement , 2011, ACM Trans. Graph..

[12]  Angel X. Chang,et al.  Deep convolutional priors for indoor scene synthesis , 2018, ACM Trans. Graph..

[13]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Igor G. Olaizola,et al.  HTML5-based System for Interoperable 3D Digital Home Applications , 2012 .

[15]  Yan Gao,et al.  Web3D-based automatic furniture layout system using recursive case-based reasoning and floor field , 2018, Multimedia Tools and Applications.

[16]  Luc Van Gool,et al.  Procedural modeling of buildings , 2006, ACM Trans. Graph..

[17]  Jinyuan Jia,et al.  A learning framework for shape retrieval based on multilayer perceptrons , 2019, Pattern Recognit. Lett..

[18]  Elisabetta Farella,et al.  A SmartPen for 3D interaction and sketch-based surface modeling , 2015 .

[19]  Wim Lamotte,et al.  Remote rendering solutions using web technologies , 2015, Multimedia Tools and Applications.

[20]  Pascal Müller,et al.  Procedural modeling of cities , 2001, SIGGRAPH.

[21]  Shi-Min Hu,et al.  Sketch2Scene: sketch-based co-retrieval and co-placement of 3D models , 2013, ACM Trans. Graph..

[22]  Tong Lu,et al.  A new recognition model for electronic architectural drawings , 2005, Comput. Aided Des..

[23]  Pat Hanrahan,et al.  Example-based synthesis of 3D object arrangements , 2012, ACM Trans. Graph..

[24]  Zahabidin Jupri,et al.  A Comparative Study on Extraction and Recognition Method of CAD Data from CAD Drawings , 2009, 2009 International Conference on Information Management and Engineering.

[25]  Shaogang Gong,et al.  Free-hand sketch recognition by multi-kernel feature learning , 2015, Comput. Vis. Image Underst..

[26]  Tetsuo Tomiyama,et al.  Supporting conceptual design based on the function-behavior-state modeler , 1996, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[27]  Keisuke Kameyama,et al.  Towards making thinning algorithms robust against noise in sketch images , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[28]  Jiajun Wu,et al.  Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling , 2016, NIPS.

[29]  Tao Xiang,et al.  Sketch-a-Net: A Deep Neural Network that Beats Humans , 2017, International Journal of Computer Vision.

[30]  Ravin Balakrishnan,et al.  EverybodyLovesSketch: 3D sketching for a broader audience , 2009, UIST '09.

[31]  Bin Wang,et al.  Wall grid structure for interior scene synthesis , 2015, Comput. Graph..