Learning shape retrieval from different modalities

New shape retrieval framework using queries of different modalities is proposed.Kernel function computed from 3D shape similarity is used to build a common space.CNNs are used to embed three different entities into a common space.A novel 3D shape descriptor based on local CNN features is proposed.We demonstrate the performance of our framework using different benchmarks. We propose in this paper a new framework for 3D shape retrieval using queries of different modalities, which can include 3D models, images and sketches. The main scientific challenge is that different modalities have different representations and thus lie in different spaces. Moreover, the features that can be extracted from 2D images or 2D sketches are often different from those that can be computed from 3D models. Our solution is a new method based on Convolutional Neural Networks (CNN) that embeds all these entities into a common space. We propose a novel 3D shape descriptor based on local CNN features encoded using vectors of locally aggregated descriptors instead of conventional global CNN. Using a kernel function computed from 3D shape similarity, we build a target space in which wild images and sketches can be projected via two different CNNs. With this construction, matching can be performed in the common target space between same entities (sketchsketch, imageimage and 3D shape3D shape) and more importantly across different entities (sketch-image, sketch-3D shape and image-3D shape). We demonstrate the performance of the proposed framework using different benchmarks including large scale SHREC 3D datasets.

[1]  Xiang Bai,et al.  Shape Vocabulary: A Robust and Efficient Shape Representation for Shape Matching , 2014, IEEE Transactions on Image Processing.

[2]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[3]  Kun Zhou,et al.  A Survey on Partial Retrieval of 3D Shapes , 2013, Journal of Computer Science and Technology.

[4]  Hans-Peter Kriegel,et al.  State-of-the-Art in Content-Based Image and Video Retrieval , 2001, Computational Imaging and Vision.

[5]  Mohammed Bennamoun,et al.  A novel feature representation for automatic 3D object recognition in cluttered scenes , 2016, Neurocomputing.

[6]  Arjan Kuijper,et al.  Sketch-based 3D model retrieval using diffusion tensor fields of suggestive contours , 2010, ACM Multimedia.

[7]  Bo Li,et al.  SHREC'13 Track: Large Scale Sketch-Based 3D Shape Retrieval , 2013, 3DOR@Eurographics.

[8]  Ghassan Hamarneh,et al.  A Survey on Shape Correspondence , 2011, Comput. Graph. Forum.

[9]  Olivier Colot,et al.  A parts-based approach for automatic 3D shape categorization using belief functions , 2013, TIST.

[10]  Arjan Kuijper,et al.  View-based 3D model retrieval using compressive sensing based classification , 2011, 2011 7th International Symposium on Image and Signal Processing and Analysis (ISPA).

[11]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[12]  Jianxiong Xiao,et al.  3D ShapeNets: A deep representation for volumetric shapes , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Bo Li,et al.  A comparison of methods for sketch-based 3D shape retrieval , 2014, Comput. Vis. Image Underst..

[14]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[15]  Masaki Aono,et al.  Multi-Fourier spectra descriptor and augmentation with spectral clustering for 3D shape retrieval , 2009, The Visual Computer.

[16]  Andrew Zisserman,et al.  Video Google: a text retrieval approach to object matching in videos , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[17]  Satoshi Kanai Content-based 3D mesh model retrieval from hand-written sketch , 2008 .

[18]  Bin Fang,et al.  A comparison of 3D shape retrieval methods based on a large-scale benchmark supporting multimodal queries , 2015, Comput. Vis. Image Underst..

[19]  Marc'Aurelio Ranzato,et al.  DeViSE: A Deep Visual-Semantic Embedding Model , 2013, NIPS.

[20]  Masaki Aono,et al.  3D shape retrieval from a 2D image as query , 2012, Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference.

[21]  Fang Wang,et al.  Sketch-based 3D shape retrieval using Convolutional Neural Networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  Leonidas J. Guibas,et al.  Joint embeddings of shapes and images via CNN image purification , 2015, ACM Trans. Graph..

[23]  Svetlana Lazebnik,et al.  Multi-scale Orderless Pooling of Deep Convolutional Activation Features , 2014, ECCV.

[24]  Remco C. Veltkamp,et al.  A survey of content based 3D shape retrieval methods , 2004, Proceedings Shape Modeling Applications, 2004..

[25]  Petros Daras,et al.  A 3D Shape Retrieval Framework Supporting Multimodal Queries , 2010, International Journal of Computer Vision.

[26]  Larry S. Davis,et al.  Exploiting local features from deep networks for image retrieval , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[27]  Longin Jan Latecki,et al.  3D Shape Matching via Two Layer Coding , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Petros Drineas,et al.  On the Nyström Method for Approximating a Gram Matrix for Improved Kernel-Based Learning , 2005, J. Mach. Learn. Res..

[29]  Jun Qin,et al.  Content based 3D model retrieval: A survey , 2008, 2008 International Workshop on Content-Based Multimedia Indexing.

[30]  Bo Li,et al.  Sketch-based 3D model retrieval by incorporating 2D-3D alignment , 2012, Multimedia Tools and Applications.

[31]  Brian C. Lovell,et al.  Improved Image Set Classification via Joint Sparse Approximated Nearest Subspaces , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[32]  Afzal Godil,et al.  Visual Similarity Based 3D Shape Retrieval Using Bag-of-Features , 2010, 2010 Shape Modeling International Conference.

[33]  Olivier Colot,et al.  3D-Shape Retrieval Using Curves and HMM , 2010, 2010 20th International Conference on Pattern Recognition.

[34]  Nitish Srivastava,et al.  Multimodal learning with deep Boltzmann machines , 2012, J. Mach. Learn. Res..

[35]  Cordelia Schmid,et al.  Aggregating Local Image Descriptors into Compact Codes , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[36]  Paul Suetens,et al.  SHREC '11 Track: Shape Retrieval on Non-rigid 3D Watertight Meshes , 2011, 3DOR@Eurographics.

[37]  Marc Alexa,et al.  SHREC'12 Track: Sketch-Based 3D Shape Retrieval , 2012, 3DOR@Eurographics.

[38]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[39]  Hamid Laga,et al.  Covariance Descriptors for 3D Shape Matching and Retrieval , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[40]  Matthias W. Seeger,et al.  Using the Nyström Method to Speed Up Kernel Machines , 2000, NIPS.

[41]  Bernhard Schölkopf,et al.  The Kernel Trick for Distances , 2000, NIPS.

[42]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  Leonidas J. Guibas,et al.  Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[44]  Marc Alexa,et al.  Sketch-based shape retrieval , 2012, ACM Trans. Graph..

[45]  Ryutarou Ohbuchi,et al.  SHREC'12 Track: Generic 3D Shape Retrieval , 2012, 3DOR@Eurographics.

[46]  G. Gravier,et al.  Content Based Multimedia Indexing , 2019 .

[47]  Fabio Roli,et al.  Neural shape codes for 3D model retrieval , 2015, Pattern Recognit. Lett..

[48]  Adam Finkelstein,et al.  Suggestive contours for conveying shape , 2003, ACM Trans. Graph..

[49]  Olivier Colot,et al.  Local visual patch for 3d shape retrieval , 2010, 3DOR '10.

[50]  Leonidas J. Guibas,et al.  Shape google: Geometric words and expressions for invariant shape retrieval , 2011, TOGS.

[51]  Rongrong Ji,et al.  Learning High-Level Feature by Deep Belief Networks for 3-D Model Retrieval and Recognition , 2014, IEEE Transactions on Multimedia.

[52]  Longin Jan Latecki,et al.  GIFT: A Real-Time and Scalable 3D Shape Search Engine , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[53]  Tobias Schreck,et al.  Sketch-based 3D Model Retrieval using Keyshapes for Global and Local Representation , 2012, 3DOR@Eurographics.

[54]  Hamid Laga,et al.  Covariance-Based Descriptors for Efficient 3D Shape Matching, Retrieval, and Classification , 2015, IEEE Transactions on Multimedia.

[55]  Andrea Fusiello,et al.  Visual Vocabulary Signature for 3D Object Retrieval and Partial Matching , 2009, 3DOR@Eurographics.

[56]  Juhan Nam,et al.  Multimodal Deep Learning , 2011, ICML.