The assessment of 3D model representation for retrieval with CNN-RNN networks

In this paper, we propose a novel method for assessing 3D model representation via CNN and RNN networks. First, a visual tool developed with OpenGL is utilized to extract virtual views of each 3D model from different angles. These views are extracted by 10-degree wrap around the model. Second, a CNN model is used to extract the feature vectors of these virtual images. Then, these feature vectors as the input of an RNN are fused into a new feature to represent the 3D model. Finally, the Euclidean distance is used to obtain the similarity measure between two different models for the retrieval problem. In the experimental section, NTU, PSB and ShapeNet datasets are utilized to evaluate the performance of the proposed method. Several classic 3D model retrieval and classification methods are leveraged as comparison methods in this paper. The corresponding experiments also demonstrate the superiority of our approach.

[1]  Sebastian Scherer,et al.  VoxNet: A 3D Convolutional Neural Network for real-time object recognition , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[2]  Ioannis Pratikakis,et al.  Efficient 3D shape matching and retrieval using a concrete radialized spherical projection representation , 2007, Pattern Recognit..

[3]  Weizhi Nie,et al.  Convolutional deep learning for 3D object retrieval , 2017, Multimedia Systems.

[4]  Jianxiong Xiao,et al.  3D ShapeNets: A deep representation for volumetric shapes , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Min-Chun Hu,et al.  Real-Time Human Movement Retrieval and Assessment With Kinect Sensor , 2015, IEEE Transactions on Cybernetics.

[6]  Szymon Rusinkiewicz,et al.  Rotation Invariant Spherical Harmonic Representation of 3D Shape Descriptors , 2003, Symposium on Geometry Processing.

[7]  Longin Jan Latecki,et al.  GIFT: A Real-Time and Scalable 3D Shape Search Engine , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Hao Su,et al.  3D attention-driven depth acquisition for object identification , 2016, ACM Trans. Graph..

[9]  Hong Liu,et al.  View-based 3D object retrieval with discriminative views , 2017, Neurocomputing.

[10]  Ralph R. Martin,et al.  Canonical Forms for Non-Rigid 3D Shape Retrieval , 2015, 3DOR@Eurographics.

[11]  Taku Komura,et al.  Topology matching for fully automatic similarity estimation of 3D shapes , 2001, SIGGRAPH.

[12]  Bernard Chazelle,et al.  Shape distributions , 2002, TOGS.

[13]  Mohamed Daoudi,et al.  A Bayesian 3-D Search Engine Using Adaptive Views Clustering , 2007, IEEE Transactions on Multimedia.

[14]  Federico Tombari,et al.  Unique shape context for 3d data description , 2010, 3DOR '10.

[15]  Song Bai,et al.  Triplet-Center Loss for Multi-view 3D Object Retrieval , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[16]  Robert C Coghill,et al.  Voxel-based morphometry and arterial spin labeling fMRI reveal neuropathic and neuroplastic features of brain processing of itch in end-stage renal disease. , 2014, Journal of neurophysiology.

[17]  Ioannis Pratikakis,et al.  PANORAMA: A 3D Shape Descriptor Based on Panoramic Views for Unsupervised 3D Object Retrieval , 2010, International Journal of Computer Vision.

[18]  Hongxun Yao,et al.  View-based 3D object retrieval via multi-modal graph learning , 2015, Signal Process..

[19]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[20]  Yue Gao,et al.  3D model retrieval using weighted bipartite graph matching , 2011, Signal Process. Image Commun..

[21]  Peter E. Hart,et al.  Nearest neighbor pattern classification , 1967, IEEE Trans. Inf. Theory.

[22]  Sven J. Dickinson,et al.  Skeleton based shape matching and retrieval , 2003, 2003 Shape Modeling International..

[23]  Yue Gao,et al.  Camera Constraint-Free View-Based 3-D Object Retrieval , 2012, IEEE Transactions on Image Processing.

[24]  Stéphane Marchand-Maillet,et al.  Classification and Retrieval of Archaeological Potsherds Using Histograms of Spherical Orientations , 2016, ACM Journal on Computing and Cultural Heritage.

[25]  Yue Gao,et al.  Multi-Modal Clique-Graph Matching for View-Based 3D Model Retrieval , 2016, IEEE Transactions on Image Processing.

[26]  Daniel Cremers,et al.  A simple and effective relevance-based point sampling for 3D shapes , 2015, Pattern Recognit. Lett..

[27]  Karthik Ramani,et al.  Deep Learning 3D Shape Surfaces Using Geometry Images , 2016, ECCV.

[28]  Ioannis Pratikakis,et al.  Exploiting the PANORAMA Representation for Convolutional Neural Network Classification and Retrieval , 2017, 3DOR@Eurographics.

[29]  Daniel Cremers,et al.  Dense Non-rigid Shape Correspondence Using Random Forests , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  Ioannis Pratikakis,et al.  Ensemble of PANORAMA-based convolutional neural networks for 3D model classification and retrieval , 2017, Comput. Graph..

[31]  Yi Yang,et al.  Beyond Doctors: Future Health Prediction from Multimedia and Multimodal Observations , 2015, ACM Multimedia.

[32]  Ming Ouhyoung,et al.  On Visual Similarity Based 3D Model Retrieval , 2003, Comput. Graph. Forum.

[33]  Chang-Hsing Lee,et al.  A new 3D model retrieval approach based on the elevation descriptor , 2007, Pattern Recognit..

[34]  Tosiyasu L. Kunii,et al.  Constructing a Reeb graph automatically from cross sections , 1991, IEEE Computer Graphics and Applications.

[35]  Jiajun Wu,et al.  Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling , 2016, NIPS.

[36]  Bernt Schiele,et al.  Analyzing appearance and contour based methods for object categorization , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[37]  William C. Regli,et al.  Using shape distributions to compare solid models , 2002, SMA '02.

[38]  Deyu Wang,et al.  Group-Pair Convolutional Neural Networks for Multi-View Based 3D Object Retrieval , 2018, AAAI.

[39]  Yuting Su,et al.  Graph-based characteristic view set extraction and matching for 3D model retrieval , 2015, Inf. Sci..

[40]  Meng Wang,et al.  Oracle in Image Search: A Content-Based Approach to Performance Prediction , 2012, TOIS.

[41]  Marc Rioux,et al.  Description of shape information for 2-D and 3-D objects , 2000, Signal Process. Image Commun..

[42]  Zhichao Zhou,et al.  DeepPano: Deep Panoramic Representation for 3-D Shape Recognition , 2015, IEEE Signal Processing Letters.

[43]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[44]  Ron Kimmel,et al.  On Bending Invariant Signatures for Surfaces , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  Rik W. De Doncker,et al.  Packaging for power semiconductors based on the 3D printing technology Selective Laser Melting , 2014, 2014 16th European Conference on Power Electronics and Applications.

[46]  Yosi Keller,et al.  Scale-Invariant Features for 3-D Mesh Models , 2012, IEEE Transactions on Image Processing.

[47]  Subhransu Maji,et al.  Multi-view Convolutional Neural Networks for 3D Shape Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[48]  David P. Dobkin,et al.  A search engine for 3D models , 2003, TOGS.

[49]  Yue Gao,et al.  3D model comparison using spatial structure circular descriptor , 2010, Pattern Recognit..

[50]  Jitendra Malik,et al.  Recognizing Objects in Range Data Using Regional Point Descriptors , 2004, ECCV.

[51]  Gholamreza Haffari,et al.  A Latent Variable Recurrent Neural Network for Discourse Relation Language Models , 2016 .

[52]  Zhang Xiong,et al.  3D Object retrieval based on viewpoint segmentation , 2015, Multimedia Systems.