Convolutional Hypercomplex Embeddings for Link Prediction

Knowledge graph embedding research has mainly focused on the two smallest normed division algebras, R and C. Recent results suggest that trilinear products of quaternionvalued embeddings can be a more effective means to tackle link prediction. In addition, models based on convolutions on real-valued embeddings often yield state-of-the-art results for link prediction. In this paper, we investigate a composition of convolution operations with hypercomplex multiplications. We propose the four approaches QMult, OMult, ConvQ and ConvO to tackle the link prediction problem. QMult and OMult can be considered as quaternion and octonion extensions of previous state-of-the-art approaches, including DistMult and ComplEx. ConvQ and ConvO build upon QMult and OMult by including convolution operations in a way inspired by the residual learning framework. We evaluated our approaches on seven link prediction datasets including WN18RR, FB15K-237, and YAGO3-10. Experimental results suggest that the benefits of learning hypercomplexvalued vector representations become more apparent as the size and complexity of the knowledge graph grows. ConvO outperforms state-of-the-art approaches on FB15K-237 in MRR, Hit@1 and Hit@3, while QMult, OMult, ConvQ and ConvO outperform stateof-the-approaches on YAGO3-10 in all metrics. Results also suggest that link prediction performance can be further improved via prediction averaging. To foster reproducible research, we provide an open-source implementation of approaches, including training and evaluation scripts as well as pretrained models.1

[1]  Timothy M. Hospedales,et al.  Hypernetwork Knowledge Graph Embeddings , 2018, ICANN.

[2]  Guanyu Li,et al.  A Quaternion-Embedded Capsule Network Model for Knowledge Graph Completion , 2020, IEEE Access.

[3]  Pasquale Minervini,et al.  Convolutional 2D Knowledge Graph Embeddings , 2017, AAAI.

[4]  Timothy M. Hospedales,et al.  Interpreting Knowledge Graph Relation Representation from Word Embeddings , 2021, ICLR.

[5]  William Rowan Hamilton,et al.  ON QUATERNIONS, OR ON A NEW SYSTEM OF IMAGINARIES IN ALGEBRA , 1847 .

[6]  Nicholas Jing Yuan,et al.  Collaborative Knowledge Base Embedding for Recommender Systems , 2016, KDD.

[7]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[8]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[9]  Dai Quoc Nguyen,et al.  A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network , 2017, NAACL.

[10]  Jianfeng Gao,et al.  Embedding Entities and Relations for Learning and Inference in Knowledge Bases , 2014, ICLR.

[11]  Christopher R'e,et al.  Low-Dimensional Hyperbolic Knowledge Graph Embeddings , 2020, ACL.

[12]  Guillaume Bouchard,et al.  Complex Embeddings for Simple Link Prediction , 2016, ICML.

[13]  Rainer Gemulla,et al.  You CAN Teach an Old Dog New Tricks! On Training Knowledge Graph Embeddings , 2020, ICLR.

[14]  Markus Krötzsch,et al.  Getting the Most Out of Wikidata: Semantic Technology Usage in Wikipedia's Knowledge Graph , 2018, SEMWEB.

[15]  Siu Cheung Hui,et al.  Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters , 2021, ICLR.

[16]  Andrew McCallum,et al.  A2N: Attending to Neighbors for Knowledge Graph Inference , 2019, ACL.

[17]  Axel-Cyrille Ngonga Ngomo,et al.  Out-of-Vocabulary Entities in Link Prediction , 2021, ArXiv.

[18]  Haonan Lu,et al.  DensE: An enhanced non-commutative representation for knowledge graph embedding with adaptive semantic hierarchy , 2022, Neurocomputing.

[19]  David Rolnick,et al.  How to Start Training: The Effect of Initialization and Architecture , 2018, NeurIPS.

[20]  Axel-Cyrille Ngonga Ngomo,et al.  Convolutional Complex Knowledge Graph Embeddings , 2021, ESWC.

[21]  Lina Yao,et al.  Quaternion Knowledge Graph Embeddings , 2019, NeurIPS.

[22]  Philip S. Yu,et al.  A Survey on Knowledge Graphs: Representation, Acquisition, and Applications , 2020, IEEE Transactions on Neural Networks and Learning Systems.

[23]  Timothy M. Hospedales,et al.  Multi-relational Poincaré Graph Embeddings , 2019, NeurIPS.

[24]  Douwe Kiela,et al.  Poincaré Embeddings for Learning Hierarchical Representations , 2017, NIPS.

[25]  Evgeniy Gabrilovich,et al.  A Review of Relational Machine Learning for Knowledge Graphs , 2015, Proceedings of the IEEE.

[26]  Jason Weston,et al.  Question Answering with Subgraph Embeddings , 2014, EMNLP.

[27]  Edward Grefenstette,et al.  Differentiable Reasoning on Large Knowledge Bases and Natural Language , 2019, Knowledge Graphs for eXplainable Artificial Intelligence.

[28]  Muhammad Saleem,et al.  Big linked cancer data: Integrating linked TCGA and PubMed , 2014, J. Web Semant..

[29]  Ying Zhang,et al.  Quaternion Convolutional Neural Networks for End-to-End Automatic Speech Recognition , 2018, INTERSPEECH.

[30]  Axel-Cyrille Ngonga Ngomo,et al.  A Physical Embedding Model for Knowledge Graphs , 2020, ArXiv.

[31]  Hans-Peter Kriegel,et al.  A Three-Way Model for Collective Learning on Multi-Relational Data , 2011, ICML.

[32]  Nicolas Usunier,et al.  Canonical Tensor Decomposition for Knowledge Base Completion , 2018, ICML.

[33]  John C. Baez,et al.  The Octonions , 2001 .

[34]  Axel-Cyrille Ngonga Ngomo,et al.  A shallow neural model for relation prediction , 2021, 2021 IEEE 15th International Conference on Semantic Computing (ICSC).

[35]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[36]  Timothy M. Hospedales,et al.  TuckER: Tensor Factorization for Knowledge Graph Completion , 2019, EMNLP.

[37]  Paul Buitelaar,et al.  Utilizing Knowledge Graphs for Neural Machine Translation Augmentation , 2019, K-CAP.

[38]  Kevin Chen-Chuan Chang,et al.  A Comprehensive Survey of Graph Embedding: Problems, Techniques, and Applications , 2017, IEEE Transactions on Knowledge and Data Engineering.

[39]  Guillaume Bouchard,et al.  Knowledge Graph Completion via Complex Tensor Factorization , 2017, J. Mach. Learn. Res..

[40]  Jian-Yun Nie,et al.  RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space , 2018, ICLR.

[41]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..