An Iterative and Incremental Data Preprocessing Procedure for Improving the Risk of Big Data Project

Big data applications can enhance the market competitive advantages of enterprises and organizations and can improve people’s quality of life. However, by the impact of many factors, failure rate of big data project is higher than the IT project. In order to reduce the risk of failure, big data projects must overcome a serial of challenges. Ambiguous requirements, poor data quality, and lacking changeability and extensity will directly affect the results of big data analytics, and even cause the wrong decision, inaccurate prediction and improper planning. Making the big data projects have potential failure risk. For this, this paper applies iterative and incremental development (IID) into the data preprocessing, draws up the iterative and incremental data quality improvement (IIDQI) procedure. Applied IIDQI procedure, iterative detects and identifies the defects of data quality, and incrementally strengthen big data quality and control the factors of failure risk. Iterative inspection activities can effectively enhance data quality, communication efficiency, and requirements precision to reduce the risk of big data project failure.