ETL Best Practices for Data Quality Checks in RIS Databases

The topic of data integration from external data sources or independent IT-systems has received increasing attention recently in IT departments as well as at management level, in particular concerning data integration in federated database systems. An example of the latter are commercial research information systems (RIS), which regularly import, cleanse, transform and prepare the analysis research information of the institutions of a variety of databases. In addition, all these so-called steps must be provided in a secured quality. As several internal and external data sources are loaded for integration into the RIS, ensuring information quality is becoming increasingly challenging for the research institutions. Before the research information is transferred to a RIS, it must be checked and cleaned up. An important factor for successful or competent data integration is therefore always the data quality. The removal of data errors (such as duplicates and harmonization of the data structure, inconsistent data and outdated data, etc.) are essential tasks of data integration using extract, transform, and load (ETL) processes. Data is extracted from the source systems, transformed and loaded into the RIS. At this point conflicts between different data sources are controlled and solved, as well as data quality issues during data integration are eliminated. Against this background, our paper presents the process of data transformation in the context of RIS which gains an overview of the quality of research information in an institution’s internal and external data sources during its integration into RIS. In addition, the question of how to control and improve the quality issues during the integration process in RIS will be addressed.

[1]  Richard Y. Wang,et al.  Data quality assessment , 2002, CACM.

[2]  Gunter Saake,et al.  Data measurement in research information systems: metrics for the evaluation of data quality , 2018, Scientometrics.

[3]  Otmane Azeroual,et al.  The Effects of Using Business Intelligence Systems on an Excellence Management and Decision-Making Process by Start-Up Companies: A Case Study , 2018, ArXiv.

[4]  Stuart E. Madnick,et al.  Overview and Framework for Data and Information Quality Research , 2009, JDIQ.

[5]  Giri Kumar Tayi,et al.  Examining data quality , 1998, CACM.

[6]  Diane M. Strong,et al.  Beyond Accuracy: What Data Quality Means to Data Consumers , 1996, J. Manag. Inf. Syst..

[7]  Marek Macura,et al.  Integration of Data from Heterogeneous Sources using ETL Technology , 2014, Comput. Sci..

[8]  George Papastefanatos,et al.  Metrics for the Prediction of Evolution Impact in ETL Ecosystems: A Case Study , 2012, Journal on Data Semantics.

[9]  Matteo Magnani,et al.  A Survey on Uncertainty Management in Data Integration , 2010, JDIQ.

[10]  Panos Vassiliadis A Survey of Extract-Transform-Load Technology , 2009, Int. J. Data Warehous. Min..

[11]  Gunter Saake,et al.  Analyzing data quality issues in research information systems via data profiling , 2018, Int. J. Inf. Manag..

[12]  Joachim Schöpfel,et al.  Quality Issues of CRIS Data: An Exploratory Investigation with Universities from Twelve Countries , 2019, Publ..

[13]  Thomas Redman,et al.  The impact of poor data quality on the typical enterprise , 1998, CACM.