Data Preprocessing Techniques

It is hard for raw industrial data accumulated by commonly implemented supervisory control and data acquisition (SCADA) system on-site to be directly employed to construct a prediction model, given that such data are always mixed with high level noise, missing points, and outliers due to the possible real-time database malfunction, data transformation, or maintenance. Thereby, the data preprocessing techniques have to be implemented, which usually contain anomaly data detection, data imputation, and data de-noising techniques. As for the issue of outliers, in this chapter, we introduce the anomaly detection methods based on fuzzy C means (FCM), K-nearest-neighbor (KNN), and dynamic time warping (DTW) algorithms. To tackle the missing data points problem, a series of data imputation methods are also described. After introducing the generic regression filling and expectation maximum methods, we supplement a varied window similarity measure method, the segmented shape-representation-based method, and the non-equal-length granules correlation method for industrial data imputation. With respect to the high level noise embodied in raw data, we then give an introduction to the well-known empirical mode decomposition (EMD) method. To verify the effectiveness of these methods, this chapter also provides a number of industrial case studies.