3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks

We propose a method for reconstructing 3D shapes from 2D sketches in the form of line drawings. Our method takes as input a single sketch, or multiple sketches, and outputs a dense point cloud representing a 3D reconstruction of the input sketch(es). The point cloud is then converted into a polygon mesh. At the heart of our method lies a deep, encoder-decoder network. The encoder converts the sketch into a compact representation encoding shape information. The decoder converts this representation into depth and normal maps capturing the underlying surface from several output viewpoints. The multi-view maps are then consolidated into a 3D point cloud by solving an optimization problem that fuses depth and normals across all viewpoints. Based on our experiments, compared to other methods, such as volumetric networks, our architecture offers several advantages, including more faithful reconstruction, higher output surface resolution, better preservation of topology and shape structure.

[1]  Adam Finkelstein,et al.  Where do people draw lines , 2008, SIGGRAPH 2008.

[2]  Thomas Brox,et al.  Single-view to Multi-view: Reconstructing Unseen Views with a Convolutional Network , 2015, ArXiv.

[3]  Andrew W. Fitzgibbon,et al.  What Shape Are Dolphins? Building 3D Morphable Models from 2D Images , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Karthik Ramani,et al.  Sketch-based 3D engineering part class browsing and retrieval , 2006, SBM'06.

[5]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[6]  Jiajun Wu,et al.  Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling , 2016, NIPS.

[7]  Alexei A. Efros,et al.  Geometric context from a single image , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[8]  Frédo Durand,et al.  Apparent ridges for line drawing , 2007, ACM Trans. Graph..

[9]  Shi-Min Hu,et al.  Sketch2Scene: sketch-based co-retrieval and co-placement of 3D models , 2013, ACM Trans. Graph..

[10]  Horst Bischof,et al.  OctNetFusion: Learning Depth Fusion from Data , 2017, 2017 International Conference on 3D Vision (3DV).

[11]  Zoran Popovic,et al.  The space of human body shapes: reconstruction and parameterization from range scans , 2003, ACM Trans. Graph..

[12]  Scott E. Reed,et al.  Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis , 2015, NIPS.

[13]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[14]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[15]  Alec R. Rivers,et al.  3D modeling with silhouettes , 2010, ACM Trans. Graph..

[16]  Matthew Turk,et al.  A Morphable Model For The Synthesis Of 3D Faces , 1999, SIGGRAPH.

[17]  Yizhou Yu,et al.  DeepSketch2Face , 2017, ACM Trans. Graph..

[18]  Abhinav Gupta,et al.  Designing deep networks for surface normal estimation , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Thomas A. Funkhouser,et al.  Sketch-based search and composition of 3D models , 2008, SBM'08.

[20]  Daniel G. Aliaga,et al.  Interactive sketching of urban procedural models , 2016, ACM Trans. Graph..

[21]  Subhransu Maji,et al.  Multi-view Convolutional Neural Networks for 3D Shape Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[22]  Ersin Yumer,et al.  Shape Synthesis from Sketches via Procedural Models and Convolutional Networks , 2017, IEEE Transactions on Visualization and Computer Graphics.

[23]  David P. Dobkin,et al.  A search engine for 3D models , 2003, TOGS.

[24]  Satoshi Matsuoka,et al.  Teddy: A Sketching Interface for 3D Freeform Design , 1999, SIGGRAPH Courses.

[25]  Konrad Schindler,et al.  Just Look at the Image: Viewpoint-Specific Surface Normal Prediction for Improved Multi-View Reconstruction , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Siddhartha Chaudhuri,et al.  CustomCut: On‐demand Extraction of Customized 3D Parts with 2D Sketches , 2016, Comput. Graph. Forum.

[27]  Thomas Brox,et al.  Learning to generate chairs with convolutional neural networks , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  John F. Hughes,et al.  SKETCH: An Interface for Sketching 3D Scenes , 1996, SIGGRAPH.

[29]  David L. Waltz,et al.  Understanding Line drawings of Scenes with Shadows , 1975 .

[30]  Thomas Brox,et al.  Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[31]  Alexei A. Efros,et al.  Colorful Image Colorization , 2016, ECCV.

[32]  Leonidas J. Guibas,et al.  ShapeNet: An Information-Rich 3D Model Repository , 2015, ArXiv.

[33]  Thomas Brox,et al.  Multi-view 3D Models from Single Images with a Convolutional Network , 2015, ECCV.

[34]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Manuel Menezes de Oliveira Neto,et al.  Domain transform for edge-aware image and video processing , 2011, ACM Trans. Graph..

[36]  Marc Alexa,et al.  A sketch-based interface for detail-preserving mesh editing , 2007, SIGGRAPH Courses.

[37]  Hans-Peter Seidel,et al.  Ridge-valley lines on meshes via implicit surface fitting , 2004, ACM Trans. Graph..

[38]  Hod Lipson,et al.  Optimization-based reconstruction of a 3D object from a single freehand line drawing , 1996, Comput. Aided Des..

[39]  Adrien Bousseau,et al.  True2Form: 3D curve networks from 2D sketches via selective regularization , 2014, ACM Trans. Graph..

[40]  Evangelos Kalogerakis,et al.  Eurographics Symposium on Geometry Processing 2015 Analysis and Synthesis of 3d Shape Families via Deep-learned Generative Models of Surfaces , 2022 .

[41]  Jitendra Malik,et al.  Interpreting line drawings of curved objects , 1986, International Journal of Computer Vision.

[42]  Gregory Shakhnarovich,et al.  Learning Representations for Automatic Colorization , 2016, ECCV.

[43]  Jitendra Malik,et al.  Hierarchical Surface Prediction for 3D Object Reconstruction , 2017, 2017 International Conference on 3D Vision (3DV).

[44]  Yong Jae Lee,et al.  ShadowDraw: real-time user guidance for freehand drawing , 2011, ACM Trans. Graph..

[45]  Adam Finkelstein,et al.  How well do line drawings depict shape? , 2009, SIGGRAPH '09.

[46]  Fang Wang,et al.  Sketch-based 3D shape retrieval using Convolutional Neural Networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Andrew W. Fitzgibbon,et al.  Bundle Adjustment - A Modern Synthesis , 1999, Workshop on Vision Algorithms.

[48]  Rob Fergus,et al.  Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[49]  John F. Hughes,et al.  SKETCH: an interface for sketching 3D scenes , 1996, SIGGRAPH.

[50]  Michael M. Kazhdan,et al.  Screened poisson surface reconstruction , 2013, TOGS.

[51]  Andrea Vedaldi,et al.  Texture Networks: Feed-forward Synthesis of Textures and Stylized Images , 2016, ICML.

[52]  Wenping Wang,et al.  Flow aligned surfacing of curve networks , 2015, ACM Trans. Graph..

[53]  Marc Alexa,et al.  Sketch-based shape retrieval , 2012, ACM Trans. Graph..

[54]  M. Eitz,et al.  Sketch-based 3 D shape retrieval , 2010 .

[55]  Honglak Lee,et al.  Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision , 2016, NIPS.

[56]  J J Koenderink,et al.  What Does the Occluding Contour Tell Us about Solid Shape? , 1984, Perception.

[57]  Mario Costa Sousa,et al.  Sketch-based modeling: A survey , 2009, Comput. Graph..

[58]  Ashutosh Saxena,et al.  Learning 3-D Scene Structure from a Single Still Image , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[59]  Bui Tuong Phong.  Illumination for computer generated pictures , 1975, Commun. ACM.

[60]  Silvio Savarese,et al.  3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction , 2016, ECCV.

[61]  Tinne Tuytelaars,et al.  Sketch classification and classification-driven analysis using Fisher vectors , 2014, ACM Trans. Graph..

[62]  Jitendra Malik,et al.  View Synthesis by Appearance Flow , 2016, ECCV.

[63]  Ryan Schmidt,et al.  Analytic drawing of 3D scaffolds , 2009, ACM Trans. Graph..

[64]  J. Koenderink,et al.  Surface perception in pictures , 1992, Perception & psychophysics.

[65]  Marc Alexa,et al.  Sketch-based 3D shape retrieval , 2010, SIGGRAPH '10.

[66]  Szymon Rusinkiewicz,et al.  Efficiently combining positions and normals for precise 3D geometry , 2005, ACM Trans. Graph..

[67]  David J. Kriegman,et al.  The Bas-Relief Ambiguity , 2004, International Journal of Computer Vision.

[68]  Daniel Cohen-Or,et al.  Sketch‐to‐Design: Context‐Based Part Assembly , 2012, Comput. Graph. Forum.

[69]  Jiantao Pu,et al.  A 2D Sketch-Based User Interface for 3D CAD Model Retrieval , 2005 .

[70]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[71]  Marc Levoy,et al.  Efficient variants of the ICP algorithm , 2001, Proceedings Third International Conference on 3-D Digital Imaging and Modeling.

[72]  Adam Finkelstein,et al.  Where do people draw lines? , 2008, ACM Trans. Graph..

[73]  Adam Finkelstein,et al.  Suggestive contours for conveying shape , 2003, ACM Trans. Graph..