On Inductive Abilities of Latent Factor Models for Relational Learning

Latent factor models are increasingly popular for modeling multi-relational knowledge graphs. By their vectorial nature, it is not only hard to interpret why this class of models works so well, but also to understand where they fail and how they might be improved. We conduct an experimental survey of state-of-the-art models, not towards a purely comparative end, but as a means to get insight about their inductive abilities. To assess the strengths and weaknesses of each model, we create simple tasks that exhibit first, atomic properties of binary relations, and then, common inter-relational inference through synthetic genealogies. Based on these experimental results, we propose new research directions to improve on existing models.

[1]  Christopher Potts,et al.  Recursive Neural Networks Can Learn Logical Semantics , 2014, CVSC.

[2]  Guillaume Bouchard,et al.  Complex Embeddings for Simple Link Prediction , 2016, ICML.

[3]  Jianfeng Gao,et al.  Embedding Entities and Relations for Learning and Inference in Knowledge Bases , 2014, ICLR.

[4]  Yehuda Koren,et al.  Factorization meets the neighborhood: a multifaceted collaborative filtering model , 2008, KDD.

[5]  Matthew Richardson,et al.  Markov logic networks , 2006, Machine Learning.

[6]  Scott Aaronson,et al.  Why Philosophers Should Care About Computational Complexity , 2011, Electron. Colloquium Comput. Complex..

[7]  Andrew McCallum,et al.  Generalizing to Unseen Entities and Entity Pairs with Row-less Universal Schema , 2016, EACL.

[8]  Thomas Demeester,et al.  Lifted Rule Injection for Relation Embeddings , 2016, EMNLP.

[9]  Andrew McCallum,et al.  Relation Extraction with Matrix Factorization and Universal Schemas , 2013, NAACL.

[10]  Luc De Raedt,et al.  Inductive Logic Programming: Theory and Methods , 1994, J. Log. Program..

[11]  Jason Weston,et al.  Translating Embeddings for Modeling Multi-relational Data , 2013, NIPS.

[12]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[13]  Yves Grandvalet,et al.  Combining Two And Three-Way Embeddings Models for Link Prediction in Knowledge Bases , 2016, J. Artif. Intell. Res..

[14]  Robert P. Goldman,et al.  From knowledge bases to decision models , 1992, The Knowledge Engineering Review.

[15]  Sameer Singh,et al.  Low-Dimensional Embeddings of Logic , 2014, ACL 2014.

[16]  Ruhi Sarikaya,et al.  Knowledge Graph Inference for spoken dialog systems , 2015, 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[17]  Thomas Demeester,et al.  Adversarial Sets for Regularising Neural Link Predictors , 2017, UAI.

[18]  Volker Tresp,et al.  Logistic Tensor Factorization for Multi-Relational Data , 2013, ArXiv.

[19]  Sameer Singh,et al.  Injecting Logical Background Knowledge into Embeddings for Relation Extraction , 2015, NAACL.

[20]  Edward Grefenstette,et al.  Towards a Formal Distributional Semantics: Simulating Logical Calculi with Tensors , 2013, *SEMEVAL.

[21]  Guillaume Bouchard,et al.  A Factorization Machine Framework for Testing Bigram Embeddings in Knowledgebase Completion , 2016, AKBC@NAACL-HLT.

[22]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[23]  Jason Weston,et al.  Learning Structured Embeddings of Knowledge Bases , 2011, AAAI.

[24]  P. Cameron.  Naïve set theory , 1998 .

[25]  Tim Rocktäschel,et al.  Learning Knowledge Base Inference with Neural Theorem Provers , 2016, AKBC@NAACL-HLT.

[26]  Masashi Shimbo,et al.  On the Equivalence of Holographic and Complex Embeddings for Link Prediction , 2017, ACL.

[27]  Jens Lehmann,et al.  DBpedia: A Nucleus for a Web of Open Data , 2007, ISWC/ASWC.

[28]  Luc De Raedt,et al.  Towards Combining Inductive Logic Programming with Bayesian Networks , 2001, ILP.

[29]  Jason Weston,et al.  Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks , 2015, ICLR.

[30]  Evgeniy Gabrilovich,et al.  A Review of Relational Machine Learning for Knowledge Graphs , 2015, Proceedings of the IEEE.

[31]  Leslie G. Valiant,et al.  Cryptographic Limitations on Learning Boolean Formulae and Finite Automata , 1993, Machine Learning: From Theory to Applications.

[32]  Andrew McCallum,et al.  Introduction to Statistical Relational Learning , 2007 .

[33]  Jianfeng Gao,et al.  Basic Reasoning with Tensor Product Representations , 2016, ArXiv.

[34]  Jason Weston,et al.  Irreflexive and Hierarchical Relations as Translations , 2013, ArXiv.

[35]  Maximilian Nickel,et al.  Complex and Holographic Embeddings of Knowledge Graphs: A Comparison , 2017, ArXiv.

[36]  Francesca A. Lisi,et al.  Inductive Logic Programming in Databases: From Datalog to $\mathcal{DL}+log}^{\neg\vee}$ , 2010, Theory and Practice of Logic Programming.

[37]  Guillaume Bouchard,et al.  On Approximate Reasoning Capabilities of Low-Rank Vector Spaces , 2015, AAAI Spring Symposia.

[38]  Hans-Peter Kriegel,et al.  A Three-Way Model for Collective Learning on Multi-Relational Data , 2011, ICML.

[39]  Lorenzo Rosasco,et al.  Holographic Embeddings of Knowledge Graphs , 2015, AAAI.

[40]  Lise Getoor,et al.  Learning Probabilistic Relational Models , 1999, IJCAI.

[41]  J. Chang,et al.  Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition , 1970 .

[42]  F. L. Hitchcock.  The Expression of a Tensor or a Polyadic as a Sum of Products , 1927 .

[43]  Peter Szolovits,et al.  What Is a Knowledge Representation? , 1993, AI Mag..

[44]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[45]  Peter Haddawy,et al.  Answering Queries from Context-Sensitive Probabilistic Knowledge Bases (cid:3) , 1996 .

[46]  Nicolas Le Roux,et al.  A latent factor model for highly multi-relational data , 2012, NIPS.

[47]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[48]  Seong-Bae Park,et al.  A Translation-Based Knowledge Graph Embedding Preserving Logical Property of Relations , 2016, HLT-NAACL.

[49]  Guillaume Bouchard,et al.  Knowledge Graph Completion via Complex Tensor Factorization , 2017, J. Mach. Learn. Res..

[50]  Praveen Paritosh,et al.  Freebase: a collaboratively created graph database for structuring human knowledge , 2008, SIGMOD Conference.

[51]  Sameer Singh,et al.  Towards Combined Matrix and Tensor Factorization for Universal Schema Relation Extraction , 2015, VS@HLT-NAACL.

[52]  Saso Dzeroski,et al.  Inductive Logic Programming: Techniques and Applications , 1993 .

[53]  Noga Alon,et al.  Sign rank versus VC dimension , 2015, COLT.

[54]  Nathan Linial,et al.  Complexity measures of sign matrices , 2007, Comb..

[55]  Antoine Bordes,et al.  Effective Blending of Two and Three-way Interactions for Modeling Multi-relational Data , 2014, ECML/PKDD.

[56]  Léon Bottou,et al.  From machine learning to machine reasoning , 2011, Machine Learning.

[57]  Wei Zhang,et al.  Knowledge vault: a web-scale approach to probabilistic knowledge fusion , 2014, KDD.

[58]  Girolamo Cardano,et al.  Artis magnae sive de regulis algebraicis liber unus , 2011 .

[59]  Yuji Matsumoto,et al.  Knowledge Transfer for Out-of-Knowledge-Base Entities: A Graph Neural Network Approach , 2017, ArXiv.

[60]  Mark Steedman,et al.  Combined Distributional and Logical Semantics , 2013, TACL.

[61]  Fabian M. Suchanek,et al.  Fast rule mining in ontological knowledge bases with AMIE+\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+$$\end{docu , 2015, The VLDB Journal.

[62]  David Heckerman,et al.  Probabilistic Entity-Relationship Models, PRMs, and Plate Models , 2004 .

[63]  H. Hornich.  Logik der Forschung , 1936 .

[64]  Yehuda Koren,et al.  Matrix Factorization Techniques for Recommender Systems , 2009, Computer.

[65]  Danqi Chen,et al.  Reasoning With Neural Tensor Networks for Knowledge Base Completion , 2013, NIPS.

[66]  William Yang Wang,et al.  Learning First-Order Logic Embeddings via Matrix Factorization , 2016, IJCAI.

[67]  Stephen Muggleton,et al.  Inverse entailment and progol , 1995, New Generation Computing.

[68]  Joshua B. Tenenbaum,et al.  Modelling Relational Data using Bayesian Clustered Tensor Factorization , 2009, NIPS.

[69]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.