Knowledge-Based Distant Regularization in Learning Probabilistic Models

Exploiting the appropriate inductive bias based on the knowledge of data is essential for achieving good performance in statistical machine learning. In practice, however, the domain knowledge of interest often provides information on the relationship of data attributes only distantly, which hinders direct utilization of such domain knowledge in popular regularization methods. In this paper, we propose the knowledge-based distant regularization framework, in which we utilize the distant information encoded in a knowledge graph for regularization of probabilistic model estimation. In particular, we propose to impose prior distributions on model parameters specified by knowledge graph embeddings. As an instance of the proposed framework, we present the factor analysis model with the knowledge-based distant regularization. We show the results of preliminary experiments on the improvement of the generalization capability of such model.

[1]  Katia P. Sycara,et al.  Nonnegative Matrix Tri-Factorization with Graph Regularization for Community Detection in Social Networks , 2015, IJCAI.

[2]  Kevin Chen-Chuan Chang,et al.  A Comprehensive Survey of Graph Embedding: Problems, Techniques, and Applications , 2017, IEEE Transactions on Knowledge and Data Engineering.

[3]  Fei Wang,et al.  Graph dual regularization non-negative matrix factorization for co-clustering , 2012, Pattern Recognit..

[4]  Minlie Huang,et al.  SSP: Semantic Space Projection for Knowledge Graph Embedding with Text Descriptions , 2016, AAAI.

[5]  Julien Mairal,et al.  Supervised feature selection in graphs with path coding penalties and network flows , 2012, J. Mach. Learn. Res..

[6]  Daniel L. Rubin,et al.  Inferring Generative Model Structure with Static Analysis , 2017, NIPS.

[7]  Pascal Fua,et al.  A constrained latent variable model , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[8]  Sameer Singh,et al.  Embedding Multimodal Relational Data , 2017, AKBC@NIPS.

[9]  Eric P. Xing,et al.  Grounding Topic Models with Knowledge Bases , 2016, IJCAI.

[10]  Daniel Oñoro-Rubio,et al.  Answering Visual-Relational Queries in Web-Extracted Knowledge Graphs , 2017, AKBC.

[11]  Christopher Ré,et al.  Learning the Structure of Generative Models without Labeled Data , 2017, ICML.

[12]  Jianfeng Gao,et al.  Embedding Entities and Relations for Learning and Inference in Knowledge Bases , 2014, ICLR.

[13]  Matthew Richardson,et al.  Markov logic networks , 2006, Machine Learning.

[14]  Zhendong Mao,et al.  Knowledge Graph Embedding: A Survey of Approaches and Applications , 2017, IEEE Transactions on Knowledge and Data Engineering.

[15]  Baogang Wei,et al.  Incorporating Probabilistic Knowledge into Topic Models , 2015, PAKDD.

[16]  Jens Lehmann,et al.  Incorporating Literals into Knowledge Graph Embeddings , 2018, SEMWEB.

[17]  Jude W. Shavlik,et al.  Knowledge-Based Artificial Neural Networks , 1994, Artif. Intell..

[18]  Jean-Philippe Vert,et al.  Group lasso with overlap and graph lasso , 2009, ICML '09.

[19]  Nathanael Perraudin,et al.  Fast Robust PCA on Graphs , 2015, IEEE Journal of Selected Topics in Signal Processing.

[20]  Zhiyuan Liu,et al.  Representation Learning of Knowledge Graphs with Entity Descriptions , 2016, AAAI.

[21]  Jun Zhu,et al.  Robust RegBayes: Selectively Incorporating First-Order Logic Domain Knowledge into Bayesian Models , 2014, ICML.

[22]  Junzhou Huang,et al.  Learning with structured sparsity , 2009, ICML '09.

[23]  Rui Zhang,et al.  Incorporating Knowledge Graph Embeddings into Topic Modeling , 2017, AAAI.

[24]  Kristian Kersting,et al.  Markov Logic Mixtures of Gaussian Processes: Towards Machines Reading Regression Data , 2012, AISTATS.

[25]  Daniel Oñoro-Rubio,et al.  Representation Learning for Visual-Relational Knowledge Graphs , 2017, ArXiv.

[26]  Glenn Fung,et al.  Knowledge-Based Support Vector Machine Classifiers , 2002, NIPS.

[27]  Christopher De Sa,et al.  Data Programming: Creating Large Training Sets, Quickly , 2016, NIPS.

[28]  Xiaojin Zhu,et al.  Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence A Framework for Incorporating General Domain Knowledge into Latent Dirichlet Allocation Using First-Order Logic , 2022 .

[29]  Evgeniy Gabrilovich,et al.  A Review of Relational Machine Learning for Knowledge Graphs , 2015, Proceedings of the IEEE.

[30]  Nicholas Jing Yuan,et al.  Collaborative Knowledge Base Embedding for Recommender Systems , 2016, KDD.

[31]  Achim Rettinger,et al.  Towards Holistic Concept Representations: Embedding Relational Knowledge, Visual Attributes, and Distributional Word Semantics , 2017, International Semantic Web Conference.

[32]  James R. Foulds,et al.  Latent Topic Networks: A Versatile Probabilistic Programming Framework for Topic Models , 2015, ICML.

[33]  Ning Chen,et al.  Bayesian inference with posterior regularization and applications to infinite latent SVMs , 2012, J. Mach. Learn. Res..

[34]  Luc De Raedt,et al.  Statistical Relational Artificial Intelligence: Logic, Probability, and Computation , 2016, Statistical Relational Artificial Intelligence.

[35]  Miao Fan,et al.  Distributed representation learning for knowledge graphs with entity descriptions , 2017, Pattern Recognit. Lett..

[36]  Michael E. Tipping,et al.  Probabilistic Principal Component Analysis , 1999 .

[37]  Guillaume Bouchard,et al.  On Approximate Reasoning Capabilities of Low-Rank Vector Spaces , 2015, AAAI Spring Symposia.