Fine-Grained Evaluation of Knowledge Graph Embedding Model in Knowledge Enhancement Downstream Tasks

Abstract Knowledge graph (KG) embedding models are proposed to encode entities and relations into a low-dimensional vector space, in turn, can support various machine learning models on KG completion with good performance and robustness. However, the current entity ranking protocol about KG completion cannot adequately evaluate the impacts of KG embedding models in real-world applications. However, KG embeddings are not widely used as word embeddings. An asserted powerful KG embedding model may not be effective in downstream tasks. So in this paper, we commit to finding the answers by using downstream tasks instead of entity ranking protocol to evaluate the effectiveness of KG embeddings. Specifically, we conduct comprehensive experiments on different KG embedding models in KG based question answering, recommendation and natural language processing tasks. Utilizing different genre of downstream tasks, we try to mine the characteristics of each KGE model in actual application scenarios, and provide guidance for the research of KGE models and knowledge-enhanced downstream tasks.

[1]  Nicholas Jing Yuan,et al.  Collaborative Knowledge Base Embedding for Recommender Systems , 2016, KDD.

[2]  Gerhard Weikum,et al.  Automated Template Generation for Question Answering over Knowledge Graphs , 2017, WWW.

[3]  Edward Y. Chang,et al.  Improving Sequential Recommendation with Knowledge-Enhanced Memory Networks , 2018, SIGIR.

[4]  Han Xiao,et al.  From One Point to a Manifold: Knowledge Graph Embedding for Precise Link Prediction , 2015, IJCAI.

[5]  Xiaodong He,et al.  Character-Level Question Answering with Attention , 2016, EMNLP.

[6]  Lina Yao,et al.  Learning to Hash for Efficient Search Over Incomplete Knowledge Graphs , 2019, 2019 IEEE International Conference on Data Mining (ICDM).

[7]  Geoffrey Zweig,et al.  Linguistic Regularities in Continuous Space Word Representations , 2013, NAACL.

[8]  Zhiyuan Liu,et al.  Learning Entity and Relation Embeddings for Knowledge Graph Completion , 2015, AAAI.

[9]  Xiaofei Wang,et al.  Distributed Pregel-based provenance-aware regular path query processing on RDF knowledge graphs , 2019, World Wide Web.

[10]  Xing Xie,et al.  A Survey on Knowledge Graph-Based Recommender Systems , 2020, IEEE Transactions on Knowledge and Data Engineering.

[11]  Minyi Guo,et al.  DKN: Deep Knowledge-Aware Network for News Recommendation , 2018, WWW.

[12]  Jian-Yun Nie,et al.  RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space , 2018, ICLR.

[13]  Lorenzo Rosasco,et al.  Holographic Embeddings of Knowledge Graphs , 2015, AAAI.

[14]  Jingyuan Zhang,et al.  Knowledge Graph Embedding Based Question Answering , 2019, WSDM.

[15]  Wei Zhang,et al.  Label-Free Distant Supervision for Relation Extraction via Knowledge Graph Embedding , 2018, EMNLP.

[16]  Jason Weston,et al.  Connecting Language and Knowledge Bases with Embedding Models for Relation Extraction , 2013, EMNLP.

[17]  Alexandros Kalousis,et al.  Hyperbolic Knowledge Graph Embeddings for Knowledge Base Completion , 2019, ESWC.

[18]  Jimmy J. Lin,et al.  Strong Baselines for Simple Question Answering over Knowledge Graphs with and without Neural Networks , 2017, NAACL.

[19]  Zhao Li,et al.  TransRHS: A Representation Learning Method for Knowledge Graphs with Relation Hierarchical Structure , 2020, IJCAI.

[20]  Zhen Wang,et al.  Knowledge Graph Embedding by Translating on Hyperplanes , 2014, AAAI.

[21]  Jason Weston,et al.  Large-scale Simple Question Answering with Memory Networks , 2015, ArXiv.

[22]  Meng Wang,et al.  Safe Medicine Recommendation via Medical Knowledge Graph Embedding , 2017, ArXiv.

[23]  Jason Weston,et al.  Translating Embeddings for Modeling Multi-relational Data , 2013, NIPS.

[24]  Yi Liu,et al.  A Survey on Blocking Technology of Entity Resolution , 2020, Journal of Computer Science and Technology.

[25]  Maosong Sun,et al.  ERNIE: Enhanced Language Representation with Informative Entities , 2019, ACL.

[26]  Hua Wu,et al.  An End-to-End Model for Question Answering over Knowledge Base with Cross-Attention Combining Global Knowledge , 2017, ACL.

[27]  Zhendong Mao,et al.  Knowledge Graph Embedding: A Survey of Approaches and Applications , 2017, IEEE Transactions on Knowledge and Data Engineering.