A Practical Framework for Evaluating the Quality of Knowledge Graph

Knowledge graphs have become much large and complex during past several years due to its wide applications in knowledge discovery. Many knowledge graphs were built using automated construction tools and via crowdsourcing. The graph may contain significant amount of syntax and semantics errors that great impact its quality. A low quality knowledge graph produce low quality application that is built on it. Therefore, evaluating quality of knowledge graph is necessary for building high quality applications. Many frameworks were proposed for systematic evaluation of knowledge graphs, but they are either too complex to be practical or lacking of scalability to large scale knowledge graphs. In this paper, we conducted a comprehensive study of existing frameworks and proposed a practical framework for evaluating quality on “fit for purpose” of knowledge graphs. We first selected a set of quality dimensions and their corresponding metrics based on the requirements of knowledge discovery based on knowledge graphs through systematic investigation of representative published applications. Then we recommended an approach for evaluating each metric considering its feasibility and scalability. The framework can be used for checking the essential quality requirements of knowledge graphs for serving the purpose of knowledge discovery.

[1]  Jens Lehmann,et al.  User-driven quality evaluation of DBpedia , 2013, I-SEMANTICS '13.

[2]  Peter Clark,et al.  Learning Knowledge Graphs for Question Answering through Conversational Dialog , 2015, NAACL.

[3]  Jingyuan Zhang,et al.  Knowledge Graph Embedding Based Question Answering , 2019, WSDM.

[4]  Heiko Paulheim,et al.  Knowledge graph refinement: A survey of approaches and evaluation methods , 2016, Semantic Web.

[5]  Jiawei Han,et al.  Individualized Knowledge Graph: A Viable Informatics Path to Precision Medicine , 2017, Circulation research.

[6]  Le Song,et al.  Variational Reasoning for Question Answering with Knowledge Graph , 2017, AAAI.

[7]  Jens Lehmann,et al.  Quality assessment for Linked Data: A Survey , 2015, Semantic Web.

[8]  Minyi Guo,et al.  RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems , 2018, CIKM.

[9]  Ming Gao,et al.  AgriKG: An Agricultural Knowledge Graph and Its Applications , 2019, DASFAA.

[10]  Huajun Chen,et al.  Knowledge-Driven Stock Trend Prediction and Explanation via Temporal Convolutional Network , 2019, WWW.

[11]  Richard M. Keller.  Building a Knowledge Graph for the Air Traffic Management Community , 2019, WWW.

[12]  Vasudeva Varma,et al.  ELDEN: Improved Entity Linking Using Densified Knowledge Graphs , 2018, NAACL-HLT.

[13]  Philippe Cudré-Mauroux,et al.  ActiveLink: Deep Active Learning for Link Prediction in Knowledge Graphs , 2019, WWW.

[14]  Linda C. Smith,et al.  A framework for information quality assessment , 2007 .

[15]  Xinbing Wang,et al.  AceKG: A Large-scale Knowledge Graph for Academic Data Mining , 2018, CIKM.

[16]  Jingqian Wen,et al.  Construction and application research of knowledge graph in aviation risk field , 2018 .

[17]  Tanvi Banerjee,et al.  A Knowledge Graph Framework for Detecting Traffic Events Using Stationary Cameras , 2017 .

[18]  Jens Lehmann,et al.  Neural Network-based Question Answering over Knowledge Graphs on Word and Character Level , 2017, WWW.

[19]  Jens Lehmann,et al.  Test-driven evaluation of linked data quality , 2014, WWW.

[20]  Diane M. Strong,et al.  Beyond Accuracy: What Data Quality Means to Data Consumers , 1996, J. Manag. Inf. Syst..

[21]  Lei Zou,et al.  Question Answering Over Knowledge Graphs: Question Understanding Via Template Decomposition , 2018, Proc. VLDB Endow..

[22]  Xin Luna Dong,et al.  Efficient Knowledge Graph Accuracy Evaluation , 2019, Proc. VLDB Endow..

[23]  Edgar Meij,et al.  Utilizing Knowledge Graphs for Text-Centric Information Retrieval , 2018, SIGIR.