3D Model Retrieval Based on a 3D Shape Knowledge Graph

A development of 3D construction technology has led to 3D models being applied in many fields, and the number of 3D models has exploded in recent years. Thus, 3D model retrieval has become a popular topic with many proposed approaches. However, all of the methods focus on the 3D model’s global structural descriptor design based on various deep learning networks and ignore the local structural information of the 3D model and the correlation of the local structures. In this paper, we propose a novel 3D model retrieval method based on a 3D shape knowledge graph. We first introduce the concept of a geometric word that can be utilized to assemble other 3D model. Second, we construct a 3D shape knowledge graph based on the geometric words, models and their relations. Additionally, we propose a novel graph embedding method to generate embeddings of nodes. Finally, an effective multiple entities’ retrieval method is used to handle the 3D model retrieval problem. More specifically, the 3D shape knowledge graph retains the basic structural information and saves these as a set of triples. Any 3D model can find its geometric words in a rich enough knowledge graph. It is reasonable that our approach can solve the cross-domain model retrieval problem. Our approach focuses on the structural information of 3D model and is not restricted by the database. We evaluate the proposed method on the ModelNet40 dataset for the 3D model retrieval task. Meanwhile, we also utilize the ShapeNet dataset to evaluate the performance of cross-domain retrieval task. Experimental results and comparisons with state-of-the-art methods demonstrate that our framework can achieve superior performance.

[1]  Daniel R. Figueiredo,et al.  struc2vec: Learning Node Representations from Structural Identity , 2017, KDD.

[2]  Jialie Shen,et al.  Forbidden City Great Wall Old SummerPalace Temple of Heaven Tiananmen Square Avenue of Stars Disneyland Resort Peninsular Hotel Tian Tan Budda Victoria Harbour Big Ben Buckingham Palace , 2016 .

[3]  Leonidas J. Guibas,et al.  Volumetric and Multi-view CNNs for Object Classification on 3D Data , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Yue Gao,et al.  3-D Object Retrieval and Recognition With Hypergraph Analysis , 2012, IEEE Transactions on Image Processing.

[5]  Yue Gao,et al.  PVRNet: Point-View Relation Neural Network for 3D Shape Recognition , 2018, AAAI.

[6]  Heng Tao Shen,et al.  Exploring Auxiliary Context: Discrete Semantic Transfer Hashing for Scalable Image Retrieval , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[7]  Zhichao Zhou,et al.  DeepPano: Deep Panoramic Representation for 3-D Shape Recognition , 2015, IEEE Signal Processing Letters.

[8]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[9]  Nitesh V. Chawla,et al.  metapath2vec: Scalable Representation Learning for Heterogeneous Networks , 2017, KDD.

[10]  Mohan S. Kankanhalli,et al.  Attentive Long Short-Term Preference Modeling for Personalized Product Search , 2018, ACM Trans. Inf. Syst..

[11]  Charu C. Aggarwal,et al.  Heterogeneous Network Embedding via Deep Architectures , 2015, KDD.

[12]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Yu-Ting Su,et al.  View-Based 3-D Model Retrieval: A Benchmark , 2018, IEEE Transactions on Cybernetics.

[14]  Xuegong Zhang,et al.  Kernel Nearest-Neighbor Algorithm , 2002, Neural Processing Letters.

[15]  Qi Tian,et al.  GIFT: Towards Scalable 3D Shape Retrieval , 2017, IEEE Transactions on Multimedia.

[16]  Xitong Liu,et al.  Latent entity space: a novel retrieval approach for entity-bearing queries , 2015, Information Retrieval Journal.

[17]  Rongrong Ji,et al.  Holistic CNN Compression via Low-Rank Decomposition with Knowledge Transfer , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Krisztian Balog,et al.  Entity Linking in Queries: Tasks and Evaluation , 2015, ICTIR.

[19]  Hui Fang,et al.  Entity Came to Rescue - Leveraging Entities to Minimize Risks in Web Search , 2014, TREC.

[20]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[21]  Yue Wang,et al.  Dynamic Graph CNN for Learning on Point Clouds , 2018, ACM Trans. Graph..

[22]  Ji Wan,et al.  Multi-view 3D Object Detection Network for Autonomous Driving , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Anan Liu,et al.  Multi-View Saliency Guided Deep Neural Network for 3-D Object Retrieval and Classification , 2020, IEEE Transactions on Multimedia.

[24]  Krisztian Balog,et al.  On Type-Aware Entity Retrieval , 2017, ICTIR.

[25]  Binh-Son Hua,et al.  Pointwise Convolutional Neural Networks , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[26]  Szymon Rusinkiewicz,et al.  Rotation Invariant Spherical Harmonic Representation of 3D Shape Descriptors , 2003, Symposium on Geometry Processing.

[27]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[28]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.

[29]  Ming Ouhyoung,et al.  On Visual Similarity Based 3D Model Retrieval , 2003, Comput. Graph. Forum.

[30]  Jianxiong Xiao,et al.  3D ShapeNets: A deep representation for volumetric shapes , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Jure Leskovec,et al.  node2vec: Scalable Feature Learning for Networks , 2016, KDD.

[32]  Yue Gao,et al.  Inductive Multi-Hypergraph Learning and Its Application on View-Based 3D Object Classification , 2018, IEEE Transactions on Image Processing.

[33]  Yue Gao,et al.  PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition , 2018, ACM Multimedia.

[34]  Reza Bosagh Zadeh,et al.  FusionNet: 3D Object Classification Using Multiple Data Representations , 2016, ArXiv.

[35]  Philip S. Yu,et al.  Transfer Joint Matching for Unsupervised Domain Adaptation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  Donald J. Berndt,et al.  Using Dynamic Time Warping to Find Patterns in Time Series , 1994, KDD Workshop.

[37]  Nitesh V. Chawla,et al.  Heterogeneous Graph Neural Network , 2019, KDD.

[38]  Victor S. Lempitsky,et al.  Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[39]  Philip S. Yu,et al.  Transfer Feature Learning with Joint Distribution Adaptation , 2013, 2013 IEEE International Conference on Computer Vision.

[40]  Ioannis Pratikakis,et al.  Ensemble of PANORAMA-based convolutional neural networks for 3D model classification and retrieval , 2017, Comput. Graph..

[41]  Rongrong Ji,et al.  Semi-Supervised Adversarial Monocular Depth Estimation , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Yue Gao,et al.  Multi-Modal Clique-Graph Matching for View-Based 3D Model Retrieval , 2016, IEEE Transactions on Image Processing.

[43]  James P. Callan,et al.  Query Expansion with Freebase , 2015, ICTIR.

[44]  Heng Tao Shen,et al.  Efficient Supervised Discrete Multi-View Hashing for Large-Scale Multimedia Search , 2020, IEEE Transactions on Multimedia.

[45]  Yue Gao,et al.  MLVCNN: Multi-Loop-View Convolutional Neural Network for 3D Shape Retrieval , 2019, AAAI.

[46]  Laura Dietz,et al.  ENT Rank: Retrieving Entities for Topical Information Needs through Entity-Neighbor-Text Relations , 2019, SIGIR.

[47]  Subhransu Maji,et al.  Multi-view Convolutional Neural Networks for 3D Shape Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[48]  Steven Skiena,et al.  Don't Walk, Skip!: Online Learning of Multi-scale Network Embeddings , 2016, ASONAM.