A New Approach for Missing Data Imputation in Big Data Interface

The three-stage approach for missing data imputation in Big data interface is proposed in the paper. The first stage includes designing the Big data model in the task of missing data recovery, which enables to process the structured and semistructured data. The next stage is developing the method of missing data recovery based on functional dependencies and association rules. The estimating the algorithm complexity for missing data recovery is provided at the last stage. The proposed method of missing data recovery creates additional data values using a based domain and functional dependencies and adds these values in available training data. The performing the analysis of data different types is possible too. The correctness of the imputed values is verified on the classifier built on the original dataset. The proposed method performs in 12% better than the EM and RF methods for 30% missing data and enables the parallel execution in distributed databases. The acceleration for m=41 attributes is larger in 12.5 times for 20 servers (processors) comparing with the non-parallel modeю

[1]  Tshilidzi Marwala,et al.  Missing data: A comparison of neural network and expectation maximization techniques , 2007 .

[2]  Nuryazmin Ahmat Zainuri,et al.  A comparison of various imputation methods for missing values in air quality data , 2015 .

[3]  Olena Vovk,et al.  UNCERTAINTY REDUCTION IN BIG DATA CATALOGUE FOR INFORMATION PRODUCT QUALITY EVALUATION , 2018 .

[4]  Tshilidzi Marwala,et al.  Deep Learning and Missing Data in Engineering Systems , 2018, Studies in Big Data.

[5]  Craig K Enders,et al.  A 'missing not at random' (MNAR) and 'missing at random' (MAR) growth model comparison with a buprenorphine/naloxone clinical trial. , 2015, Addiction.

[6]  Ahmet Arslan,et al.  A hybrid method for imputation of missing values using optimized fuzzy c-means with support vector regression and a genetic algorithm , 2013, Inf. Sci..

[7]  Katherine J. Lee,et al.  The rise of multiple imputation: a review of the reporting and implementation of the method in medical research , 2015, BMC Medical Research Methodology.

[8]  Leonidas Sakalauskas,et al.  Big Data Mining Using Public Distributed Computing , 2018, Inf. Technol. Control..

[9]  Ting Hsiang Lin,et al.  A comparison of multiple imputation with EM algorithm and MCMC method for quality of life missing data , 2010 .

[10]  Tra My Pham,et al.  Missing data and multiple imputation in clinical epidemiological research , 2017, Clinical epidemiology.

[11]  Wael Alzyadat,et al.  Missing Data Prediction using Correlation Genetic Algorithm and SVM Approach , 2020 .

[12]  Grigore Stamatescu,et al.  Sensing and Data-Driven Control for Smart Building and Smart City Systems , 2019, J. Sensors.

[13]  Davar Giveki,et al.  Automatic detection of erythemato-squamous diseases using PSO-SVM based on association rules , 2013, Eng. Appl. Artif. Intell..

[14]  Tshilidzi Marwala,et al.  Introduction to Missing Data Estimation , 2018, Studies in Big Data.

[15]  Lazar Rakovic,et al.  Shadow IT - Systematic Literature Review , 2020, Inf. Technol. Control..

[16]  John B Carlin,et al.  A review of the reporting and handling of missing data in cohort studies with repeated assessment of exposure measures , 2012, BMC Medical Research Methodology.

[17]  Sung Wook Baik,et al.  Bagging Ensemble of Multilayer Perceptrons for Missing Electricity Consumption Data Imputation , 2020, Sensors.

[18]  Nataliya Shakhovska,et al.  ASSOCIATION RULES MINING IN BIG DATA , 2018 .

[19]  Teresa A. Myers.  Goodbye, Listwise Deletion: Presenting Hot Deck Imputation as an Easy and Effective Tool for Handling Missing Data , 2011 .

[20]  Devvi Sarwinda,et al.  Missing Value Analysis of Numerical Data using Fractional Hot Deck Imputation , 2019, 2019 3rd International Conference on Informatics and Computational Sciences (ICICoS).

[21]  Juan M. Corchado,et al.  Investigation of the Effects of Imputation Methods for Gene Regulatory Networks Modelling Using Dynamic Bayesian Networks , 2016, DCAI.

[22]  Alberto Maydeu-Olivares,et al.  Fitting Ordinal Factor Analysis Models With Missing Data: A Comparison Between Pairwise Deletion and Multiple Imputation , 2020, Educational and psychological measurement.

[23]  Natalia Kryvinska,et al.  Recovery of Incomplete IoT Sensed Data using High-Performance Extended-Input Neural-Like Structure , 2019, EUSPN/ICTH.

[24]  David S. Siroky.  Navigating Random Forests and related advances in algorithmic modeling , 2009 .

[25]  Md Zahidul Islam,et al.  Missing value imputation using decision trees and decision forests by splitting and merging records: Two novel techniques , 2013, Knowl. Based Syst..

[26]  Alexander Kowarik,et al.  Imputation with the R Package VIM , 2016 .