Diachronic Embedding for Temporal Knowledge Graph Completion

Knowledge graphs (KGs) typically contain temporal facts indicating relationships among entities at different times. Due to their incompleteness, several approaches have been proposed to infer new facts for a KG based on the existing ones-a problem known as KG completion. KG embedding approaches have proved effective for KG completion, however, they have been developed mostly for static KGs. Developing temporal KG embedding models is an increasingly important problem. In this paper, we build novel models for temporal KG completion through equipping static models with a diachronic entity embedding function which provides the characteristics of entities at any point in time. This is in contrast to the existing temporal KG embedding approaches where only static entity features are provided. The proposed embedding function is model-agnostic and can be potentially combined with any static model. We prove that combining it with SimplE, a recent model for static KG embedding, results in a fully expressive model for temporal KG completion. Our experiments indicate the superiority of our proposal compared to existing baselines.

[1]  Siamak Ravanbakhsh,et al.  Improved Knowledge Graph Embedding using Background Taxonomic Information , 2018, AAAI.

[2]  Seyed Mehran Kazemi,et al.  SimplE Embedding for Link Prediction in Knowledge Graphs , 2018, NeurIPS.

[3]  Jian-Yun Nie,et al.  RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space , 2018, ICLR.

[4]  Zhendong Mao,et al.  Knowledge Graph Embedding: A Survey of Approaches and Applications , 2017, IEEE Transactions on Knowledge and Data Engineering.

[5]  Jure Leskovec,et al.  Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change , 2016, ACL.

[6]  Jason Weston,et al.  Translating Embeddings for Modeling Multi-relational Data , 2013, NIPS.

[7]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[8]  Martin Theobald,et al.  A Temporal-Probabilistic Database Model for Information Extraction , 2013, Proc. VLDB Endow..

[9]  Zhiyuan Liu,et al.  Learning Entity and Relation Embeddings for Knowledge Graph Completion , 2015, AAAI.

[10]  Le Song,et al.  Know-Evolve: Deep Temporal Reasoning for Dynamic Knowledge Graphs , 2017, ICML.

[11]  Tom M. Mitchell,et al.  Random Walk Inference and Learning in A Large Scale Knowledge Base , 2011, EMNLP.

[12]  Guillaume Bouchard,et al.  Knowledge Graph Completion via Complex Tensor Factorization , 2017, J. Mach. Learn. Res..

[13]  Nicolas Usunier,et al.  Canonical Tensor Decomposition for Knowledge Base Completion , 2018, ICML.

[14]  Pasquale Minervini,et al.  Convolutional 2D Knowledge Graph Embeddings , 2017, AAAI.

[15]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[16]  Timothy M. Hospedales,et al.  TuckER: Tensor Factorization for Knowledge Graph Completion , 2019, EMNLP.

[17]  Dat Quoc Nguyen An overview of embedding models of entities and relationships for knowledge base completion , 2017, ArXiv.

[18]  Luc De Raedt,et al.  ProbLog: A Probabilistic Prolog and its Application in Link Discovery , 2007, IJCAI.

[19]  Zhifang Sui,et al.  Towards Time-Aware Knowledge Graph Completion , 2016, COLING.

[20]  Volker Tresp,et al.  Embedding models for episodic knowledge graphs , 2018, J. Web Semant..

[21]  Mathias Niepert,et al.  Learning Sequence Encoders for Temporal Knowledge Graph Completion , 2018, EMNLP.

[22]  H. Stuckenschmidt,et al.  Applying Markov Logic for Debugging Probabilistic Temporal Knowledge Bases , 2014 .

[23]  Rudolf Kadlec,et al.  Knowledge Base Completion: Baselines Strike Back , 2017, Rep4NLP@ACL.

[24]  Hongyuan Zha,et al.  DyRep: Learning Representations over Dynamic Graphs , 2019, ICLR.

[25]  Lise Getoor,et al.  A short introduction to probabilistic soft logic , 2012, NIPS 2012.

[26]  Steven Skiena,et al.  Statistically Significant Detection of Linguistic Change , 2014, WWW.

[27]  Seyed Mehran Kazemi,et al.  RelNN: A Deep Neural Model for Relational Learning , 2017, AAAI.

[28]  H. S. Carslaw,et al.  Introduction to the Theory of Fourier's Series and Integrals , 1921, Nature.

[29]  A. Lapedes,et al.  Nonlinear signal processing using neural networks: Prediction and system modelling , 1987 .

[30]  Srijan Kumar,et al.  Learning Dynamic Embeddings from Temporal Interaction Networks , 2018 .

[31]  Evgeniy Gabrilovich,et al.  A Review of Relational Machine Learning for Knowledge Graphs , 2015, Proceedings of the IEEE.

[32]  Jianfeng Gao,et al.  Embedding Entities and Relations for Learning and Inference in Knowledge Bases , 2014, ICLR.

[33]  Lizhen Qu,et al.  STransE: a novel embedding model of entities and relationships in knowledge bases , 2016, NAACL.

[34]  Jiliang Tang,et al.  Streaming Graph Neural Networks , 2018, SIGIR.

[35]  Ondrej Kuzelka,et al.  Lifted Relational Neural Networks , 2015, CoCo@NIPS.

[36]  Seyed Mehran Kazemi,et al.  Bridging Weighted Rules and Graph Random Walks for Statistical Relational Models , 2018, Front. Robot. AI.

[37]  Yiming Yang,et al.  Analogical Inference for Multi-relational Embeddings , 2017, ICML.

[38]  David Poole,et al.  Negation Without Negation in Probabilistic Logic Programming , 2016, KR.

[39]  Heiner Stuckenschmidt,et al.  Marrying Uncertainty and Time in Knowledge Graphs , 2017, AAAI.

[40]  Jure Leskovec,et al.  How Powerful are Graph Neural Networks? , 2018, ICLR.

[41]  Stephan Mandt,et al.  Dynamic Word Embeddings , 2017, ICML.

[42]  F. L. Hitchcock The Expression of a Tensor or a Polyadic as a Sum of Products , 1927 .

[43]  Vít Novácek,et al.  Regularizing Knowledge Graph Embeddings via Equivalence and Inversion Axioms , 2017, ECML/PKDD.

[44]  Lorenzo Rosasco,et al.  Holographic Embeddings of Knowledge Graphs , 2015, AAAI.

[45]  Hans-Peter Kriegel,et al.  A Three-Way Model for Collective Learning on Multi-Relational Data , 2011, ICML.

[46]  Partha Talukdar,et al.  HyTE: Hyperplane-based Temporally aware Knowledge Graph Embedding , 2018, EMNLP.

[47]  Zhen Wang,et al.  Knowledge Graph Embedding by Translating on Hyperplanes , 2014, AAAI.

[48]  Danqi Chen,et al.  Reasoning With Neural Tensor Networks for Knowledge Base Completion , 2013, NIPS.

[49]  Guillaume Bouchard,et al.  Complex Embeddings for Simple Link Prediction , 2016, ICML.

[50]  Kristian Kersting,et al.  Relational Logistic Regression: The Directed Analog of Markov Logic Networks , 2014, StarAI@AAAI.

[51]  Henry A. Kautz,et al.  Slice Normalized Dynamic Markov Logic Networks , 2012, NIPS.

[52]  Henry A. Kautz,et al.  Recognizing Multi-Agent Activities from GPS Data , 2010, AAAI.

[53]  Luc De Raedt,et al.  Statistical Relational Artificial Intelligence: Logic, Probability, and Computation , 2016, Statistical Relational Artificial Intelligence.

[54]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[55]  Heiner Stuckenschmidt,et al.  Rule Based Temporal Inference , 2017, ICLP.

[56]  Wei Zhang,et al.  Knowledge vault: a web-scale approach to probabilistic knowledge fusion , 2014, KDD.

[57]  Ni Lao,et al.  Relational retrieval using a combination of path-constrained random walks , 2010, Machine Learning.

[58]  Slav Petrov,et al.  Temporal Analysis of Language through Neural Language Models , 2014, LTCSS@ACL.

[59]  Timothy M. Hospedales,et al.  Hypernetwork Knowledge Graph Embeddings , 2018, ICANN.

[60]  Andrew McCallum,et al.  Introduction to Statistical Relational Learning , 2007 .

[61]  Pascal Poupart,et al.  Relational Representation Learning for Dynamic (Knowledge) Graphs: A Survey , 2019, ArXiv.

[62]  Giambattista Parascandolo,et al.  Taming the waves: sine as activation function in deep neural networks , 2017 .