PhotoShape

Existing online 3D shape repositories contain thousands of 3D models but lack photorealistic appearance. We present an approach to automatically assign high-quality, realistic appearance models to large scale 3D shape collections. The key idea is to jointly leverage three types of online data -- shape collections, material collections, and photo collections, using the photos as reference to guide assignment of materials to shapes. By generating a large number of synthetic renderings, we train a convolutional neural network to classify materials in real photos, and employ 3D-2D alignment techniques to transfer materials to different parts of each shape model. Our system produces photorealistic, relightable, 3D shapes (PhotoShapes).

[1]  Jitendra Malik,et al.  Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach , 1996, SIGGRAPH.

[2]  David Salesin,et al.  Image Analogies , 2001, SIGGRAPH.

[3]  Alexei A. Efros,et al.  Image quilting for texture synthesis and transfer , 2001, SIGGRAPH.

[4]  Wojciech Matusik,et al.  A data-driven reflectance model , 2003, ACM Trans. Graph..

[5]  MatusikWojciech,et al.  A data-driven reflectance model , 2003 .

[6]  Sung Yong Shin,et al.  On pixel-based texture synthesis by non-parametric sampling , 2006, Comput. Graph..

[7]  Steve Marschner,et al.  Microfacet Models for Refraction through Rough Surfaces , 2007, Rendering Techniques.

[8]  Dani Lischinski,et al.  Solid texture synthesis from 2D exemplars , 2007, ACM Trans. Graph..

[9]  Lévy Bruno,et al.  What you seam is what you get , 2009 .

[10]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[11]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Antonio Torralba,et al.  SIFT Flow: Dense Correspondence across Scenes and Its Applications , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Vladlen Koltun,et al.  Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials , 2011, NIPS.

[14]  Hans-Peter Seidel,et al.  Material memex , 2012, ACM Trans. Graph..

[15]  Brent Burley.  Physically-Based Shading at Disney , 2012 .

[16]  Daniel Cohen-Or,et al.  Projective analysis for 3D shape segmentation , 2013, ACM Trans. Graph..

[17]  Jaakko Lehtinen,et al.  Practical SVBRDF capture in the frequency domain , 2013, ACM Trans. Graph..

[18]  Edward H. Adelson,et al.  Recognizing Materials Using Perceptually Inspired Features , 2013, International Journal of Computer Vision.

[19]  Noah Snavely,et al.  OpenSurfaces , 2013, ACM Trans. Graph..

[20]  Iasonas Kokkinos,et al.  Describing Textures in the Wild , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Pieter Peers,et al.  Appearance-from-motion , 2014, ACM Trans. Graph..

[22]  Manmohan Krishna Chandraker,et al.  On Shape and Material Recovery from Motion , 2014, ECCV.

[23]  Pieter Peers,et al.  genBRDF: discovering new analytic BRDFs with genetic programming , 2014, ACM Trans. Graph..

[24]  Yaser Sheikh,et al.  3D object manipulation in a single photograph using stock 3D models , 2014, ACM Trans. Graph..

[25]  Pietro Perona,et al.  The Ignorant Led by the Blind: A Hybrid Human–Machine Vision System for Fine-Grained Categorization , 2014, International Journal of Computer Vision.

[26]  E. Adelson,et al.  Accuracy and speed of material categorization in real-world images. , 2014, Journal of vision.

[27]  Jaakko Lehtinen,et al.  Two-shot SVBRDF capture for stationary materials , 2015, ACM Trans. Graph..

[28]  Noah Snavely,et al.  Material recognition in the wild with the Materials in Context Database , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Leon A. Gatys,et al.  Texture Synthesis Using Convolutional Neural Networks , 2015, NIPS.

[30]  Olga Sorkine-Hornung,et al.  Synthesis of Complex Image Appearance from Limited Exemplars , 2015, TOGS.

[31]  Ko Nishino,et al.  Automatically discovering local visual material attributes , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Vladlen Koltun,et al.  Single-view reconstruction via joint analysis of image and shape collections , 2015, ACM Trans. Graph..

[33]  Leonidas J. Guibas,et al.  Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[34]  Hang Zhang,et al.  Reflectance hashing for material recognition , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Leonidas J. Guibas,et al.  ShapeNet: An Information-Rich 3D Model Repository , 2015, ArXiv.

[36]  Subhransu Maji,et al.  Deep filter banks for texture recognition and segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Shi-Min Hu,et al.  Magic decorator , 2015, ACM Trans. Graph..

[38]  Mario Fritz,et al.  Deep Reflectance Maps , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Vladlen Koltun,et al.  Playing for Data: Ground Truth from Computer Games , 2016, ECCV.

[41]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Ko Nishino,et al.  Shape and Reflectance Estimation in the Wild , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Leonidas J. Guibas,et al.  Unsupervised texture transfer from images to model collections , 2016, ACM Trans. Graph..

[44]  Yong Yu,et al.  Sparse-as-possible SVBRDF acquisition , 2016, ACM Trans. Graph..

[45]  Alexei A. Efros,et al.  A 4D Light-Field Dataset and CNN Architectures for Material Recognition , 2016, ECCV.

[46]  Ko Nishino,et al.  Reflectance and Illumination Recovery in the Wild , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[47]  Vladlen Koltun,et al.  Photographic Image Synthesis with Cascaded Refinement Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[48]  Daniel Cohen-Or,et al.  Deep Correlations for Texture Synthesis , 2017, ACM Trans. Graph..

[49]  Xiao Li,et al.  Modeling surface appearance from a single photograph using self-augmented convolutional neural networks , 2017, ACM Trans. Graph..

[50]  Mario Fritz,et al.  Novel Views of Objects from a Single Image , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[51]  Hamid Izadinia,et al.  IM2CAD , 2016, 1608.05137.

[52]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Matthias Nießner,et al.  A Lightweight Approach for On-the-Fly Reflectance Estimation , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[54]  Luc Van Gool,et al.  Material Classification under Natural Illumination Using Reflectance Maps , 2017, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[55]  Ersin Yumer,et al.  Material Editing Using a Physically Based Rendering Network , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[56]  Hang Zhang,et al.  Differential Angular Imaging for Material Recognition , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[57]  Tobias Ritschel,et al.  Joint Material and Illumination Estimation from Photo Sets in the Wild , 2017, 2018 International Conference on 3D Vision (3DV).

[58]  Dani Lischinski,et al.  Appearance Modeling via Proxy-to-Image Alignment , 2018, ACM Trans. Graph..

[59]  Mario Fritz,et al.  Reflectance and Natural Illumination from Single-Material Specular Objects Using Deep Learning , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[60]  Roberto Cipolla,et al.  Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.