With detailed sensor and visual data from automobiles, a data-driven model can learn to classify crash-related events during a drive. We propose a neural network model accepting time-series vehicle sensor data and forward-facing videos as input for learning classification of crash-related events and varying types of such events. To elaborate, a novel recurrent neural network structure is introduced, namely, denoising gated recurrent unit with decay, in order to deal with time-series automobile sensor data with missing value and noises. Our model detects crash and near-crash events based on a large set of time-series data collected from naturalistic driving behavior. Furthermore, the model classifies those events involving pedestrians, a vehicle in front, or a vehicle on either side. The effectiveness of our model is evaluated with more than two thousand 30-s clips from naturalistic driving behavior data. The results show that the model, including sensory encoder with denoising gated recurrent unit with decay, visual encoder, and attention mechanism, outperforms gated recurrent unit with decay, gated CNN, and other baselines not only in event classification and but also in event-type classification.

Transferring and processing huge amounts of data in the cloud can violate the low latency requirements of modern IoT applications, considering underlying network infrastructure limitations. Edge data analytics is a promising solution. However, edge resources have usually less computational capabilities than cloud nodes, resulting in a higher failure rate of IoT systems. Consequently, near-real-time decisions are often based on limited and incomplete data. State-of-the-art solutions, such as operational/workload flows, data reduction, reconstruction, focus mostly on resource and network optimization, while approaches for incomplete data recovery employ a single specific method, despite diverse data characteristics. Data quality impact on accuracy of the decision-making processes is often neglected. We propose EDMFrame, a framework featuring a generic mechanism for recovery of multiple gaps in incomplete datasets, using single-technique recovery (STR) and multiple-technique recovery (MTR) involving projection recovery maps (PRMs). We further devise an adaptive storage management mechanism for reducing data stored at the edge, keeping only the data necessary for predictive analytics. We conduct experiments using time series from smart buildings, (i) automatically recovering various multiple gaps and reducing errors up to 65.48 percent with MTR compared to STR; (ii) reducing amounts of data stored to 39.9 percent on average, keeping prediction accuracy around 98.83 percent.

To mitigate the greenhouse gas effect, accurate and precise landfill gas prediction models are required for more precise prediction of the amount and recovery time of methane gas from landfills. When the study associates to greenhouse gas emissions problems, time series prediction models are of considerable interests, in which significant past records of gas data are required. This study is the first to specially impute the missing methane (CH4) data for applying in time series artificial neural network (ANN) model in an attempt to predict daily CH4 generation rate from a landfill in Regina, SK, Canada. Pre-processing was conducted on data to evaluate independent and significant meteorological input variables and provide suitable dataset for developing CH4 generation models. A two-stage time series model proposed in this study was performed by missing data imputation at the first stage, followed by a neural network auto-regressive model with exogenous inputs (NARX) at the second stage. The model with 3 layers, 5 climatic variables and 9 neurons in the hidden layer was the optimal structure. This model shows the high performance in CH4 prediction with the average index of agreement of 0.92 and the average mean absolute percentage error (MAPE) of 3.03% during the testing stage. Missing data imputation coupled with NARX method decreased the mean squared error (MSE) of the model by 84% (compared to Multilayer Perceptrons neural network model) in the testing period representing the effectiveness of missing data estimation coupling with time series ANN models in daily CH4 generation prediction.

Time series have been used in several applications such as process control, environment monitoring, financial analysis and scientific researches. However, in the presence of missing data, this study may become more complex due to a strong break of correlation among samples. Therefore, this work proposes an imputation method for time series using Genetic Programming (GP) and Lagrange Interpolation. The heuristic adopted builds an interpretable regression model that explores time series statistical features such as mean, variance and auto-correlation. It also makes use of interrelation among multivariate time series to estimate missing values. Results show that the proposed method is promising, being capable of imputing data without loosing the dataset's statistical properties, as well as allowing a better understanding of the missing data pattern from the obtained interpretable model.

Time Series Analysis has been used in several areas, such as pattern recognition, machine learning, data mining, andforecasting. However, before applying any method to model a timeseries, it is important to preprocess its observations aiming at, forexample, removing outliers, normalizing its values, and dealingwith missing data. In this context, this work proposes a newpreprocessing method to replace missing data with valid values, respecting the general time series behavior. In our experiments, we demonstrate BMDE provides impressive results, reducingthe memory usage and the complexity by analyzing a numbersignificantly lower of implicit components. The most commonlyused method to replace missing data is SSA (Singular SpectrumAnalysis). However, its memory usage limits its adoption in bigtemporal data. To overcome this situation, we present in thismanuscript a new gap filling method called Bidirectional MeanDistance Estimation (BMDE).

This thesis investigates the effectiveness of applying recurrent neural networks (RNN) to detect communication errors between radio devices, also known as supervision violations, on imbalanced data. The task is to classify whether a supervision violation is to occur within seven days. The available data is in the form of radio packets, which are being re-sampled and pre-processed such that they can be interpreted by RNNs. The RNNs are trained as both classifiers and generators to enable detection of supervision violations. Using RNNs, an extensive evaluation is made into different pre-processing methods, using multiple test sets and network architectures, applying average precision as metric and precision-recall curves as the main evaluation technique. The results show that it is possible to achieve an average precision of 0.75, and that experimentation with pre-processing parameters along with multiple testsets are needed to ensure a generalised model.

This paper investigates the jointly use of local polynomials and feedforward neural networks for estimating the probability of exceedance of the daily average for \(PM_{10}\) in the presence of missing data. In contrast to other approaches focusing on some assumption on the distribution of \(PM_{10}\), the reconstruction of the unobserved time series is obtained by using a procedure involving two nonparametric steps based on the estimation of the trend-cycle and of the superimposed nonlinear stochastic component of the series. By using Neural Network Sieve Bootstrap, the probability to overcross the limit established by the European Union for \(PM_{10}\) is evaluated at the dates where time series shows missing values. An application to real data is also presented and discussed.

There is a growing interest in mining and handling of big data, which has been rapidly accumulating in the repositories of bioprocess industries. Biopharmaceutical industries are no exception; the implementation of advanced process control strategies based on multivariate monitoring techniques in biopharmaceutical production gave rise to the generation of large amounts of data. Real-time measurements of critical quality and performance attributes collected during production can be highly useful to understand and model biopharmaceutical processes. Data mining can facilitate the extraction of meaningful relationships pertaining to these bioprocesses, and predict the performance of future cultures. This review evaluates the suitability of various metaheuristic methods available for data pre-processing, which would involve the handling of missing data, the visualisation of the data, and dimension reduction; and for data processing, which would focus on modelling of the data and the optimisation of these models in the context of biopharmaceutical process development. The advantages and the associated challenges of employing different methodologies in pre-processing and processing of the data are discussed. In light of these evaluations, a summary guideline is proposed for handling and analysis of the data generated in biopharmaceutical process development.

The estimation of a-day forward short-term electrical consumption in this research is using time series daily data from PT Perusahaan Listrik Negara (PLN) in 5-year periods. This research proposed Recurrent Neural Network and Long Shot Term Memory (LSTM) as model to forecast the electrical load using various hidden layers consist of one, two, and three layers. In this research, secondary data pre-processing is going to be done with add empty data, remove double data, and remove time with random interval. The electrical load data for 5 years is divided into 2 types of datasets, namely training data and test data. The raining data using data consist of 4 years electrical load from 2013-2016, while the test data uses data in 2017. LSTM Model then compared with Random Forest (RF) and Support Vector Machine (SVM). From the experimental result, the Root Mean Square Error (RSME) for LSTM model with 2 has the lowest compare to SVM and RF.

The development of an intelligent artificial satellite health monitoring system is a key issue in aerospace engineering that determines satellite health status and failure using telemetry data. The modern design of data mining and machine learning technologies allows the use of satellite telemetry data and the mining of integrated information to produce an advanced health monitoring system. This paper reviews the current status and presents a framework of necessary processes on data mining to solving various problems in telemetry data such as error detection, prediction, summarization, and visualization of large quantities, and help them understand the health status of the satellite and detect the symptoms of anomalies. Machine learning technologies that include neural networks, fuzzy sets, rough sets, support vector machines, Naive Bayesian, swarm optimization, and deep learning are also presented. Also, this paper reviews a wide range of existing satellite health monitoring solutions and discusses them in the framework of remote data mining techniques. In addition, we are discussing the analysis of space debris flow analysis and the prediction of low earth orbit collision based on our orbital Petri nets model. Challenges to be addressed and future directions of research are identified and an extensive bibliography is also included.

The air quality measurement data obtained from the continuous ambient air quality monitoring (CAAQM) station usually contained missing data. The missing observations of the data usually occurred due to machine failure, routine maintenance and human error. In this study, the hourly monitoring data of CO, O3, PM10, SO2, NOx, NO2, ambient temperature and humidity were used to evaluate four imputation methods (Mean Top Bottom, Linear Regression, Multiple Imputation and Nearest Neighbour). The air pollutants observations were simulated into four percentages of simulated missing data i.e. 5%, 10%, 15% and 20%. Performance measures namely the Mean Absolute Error, Root Mean Squared Error, Coefficient of Determination and Index of Agreement were used to describe the goodness of fit of the imputation methods. From the results of the performance measures, Mean Top Bottom method was selected as the most appropriate imputation method for filling in the missing values in air pollutants data.

The Internet of Things (IoT) has attracted significant attention from both academia and industry, thanks to applications such as smart cities, smart buildings and intelligent traffic management. These systems rely on data, collected from IoT devices, that are sent to the cloud for analytics. Data are either used for near real-time decisions or stored for long-term analysis. However, in highly distributed IoT systems, missing or invalid data may appear because of different reasons including sensor failures, monitoring system failures and network failures. Analyzing incomplete datasets can lead to inaccurate results and imprecise decisions, with negative effects on the target systems. Also, due to the increasing size of such systems and the consequently increasing amount of data generated from sensors, recovery of incomplete datasets for analytics on the cloud is often infeasible, due to the limited bandwidth available and the strict latency constraints of IoT applications. We propose a novel semi-automatic recursive mechanism for recovery of incomplete datasets on the edge that is closer to the source of data. This mechanism enables efficient recovery of incomplete datasets employing different forecasting techniques for multiple gaps, based on user specifications. We evaluate our approach on datasets coming from the context of smart buildings and smart homes. The experimental results show that our approach is able to identify multiple gaps, then recover incomplete datasets, decreasing forecasting error by up to 82.68%, and reducing running time by up to 52.38%.

Reliable estimation of long-range dependence parameters is vital in time series. For example, in environmental and climate science such estimation is often key to understanding climate dynamics, variability and often prediction. The challenge of data collection in such disciplines means that, in practice, the sampling pattern is either irregular or blighted by missing observations. Unfortunately, virtually all existing Hurst parameter estimation methods assume regularly sampled time series and require modification to cope with irregularity or missing data. However, such interventions come at the price of inducing higher estimator bias and variation, often worryingly ignored. This article proposes a new Hurst exponent estimation method which naturally copes with data sampling irregularity. The new method is based on a multiscale lifting transform exploiting its ability to produce wavelet-like coefficients on irregular data and, simultaneously, to effect a necessary powerful decorrelation of those coefficients. Simulations show that our method is accurate and effective, performing well against competitors even in regular data settings. Armed with this evidence our method sheds new light on long-memory intensity results in environmental and climate science applications, sometimes suggesting that different scientific conclusions may need to be drawn.

Random-variable-valued measurements (RVVMs) are proposed as a new framework for treating measurement processes that generate non-deterministic sample data. They operate by assigning a probability measure to each observed sample instantiation of a global measurement process for some particular random quantity of interest, thus allowing for the explicit quantification of response process error. Common methodologies to date treat only measurement processes that generate fixed values for each sample unit, thus generating full (though possibly inaccurate) information on the random quantity of interest. However, many applied research situations in the non-experimental sciences naturally contain response process error, e.g. when psychologists assess patient agreement with various diagnostic survey items or when conservation biologists perform formal assessments to classify species-at-risk. Ignoring the sample-unit-level uncertainty of response process error in such measurement processes can greatly compromise the quality of resulting inferences. In this paper, a general theory of RVVMs is proposed to handle response process error, and several applications are considered.

Public health risks arising from airborne pollutants, <italic>e.g</italic>., Total Suspended Particulate (<italic>TSP</italic>) matter, can significantly elevate ongoing and future healthcare costs. The chaotic behaviour of air pollutants posing major difficulties in tracking their three-dimensional movements over diverse temporal domains is a significant challenge in designing practical air quality systems. This research paper builds a deep learning hybrid <italic>CLSTM</italic> model where convolutional neural network (<italic>CNN</italic>) is amalgamed with the long short-term memory (<italic>LSTM</italic>) network to forecast hourly <italic>TSP</italic>. The <italic>CNN</italic> model entails a data processer including feature extractors that draw upon statistically significant antecedent lagged predictor variables, whereas the <italic>LSTM</italic> model encapsulates a new feature mapping scheme to predict the next hourly <italic>TSP</italic> value. The hybrid <italic>CLSTM</italic> model is comprehensively benchmarked and is seen to outperform an ensemble of five machine learning models. The efficacy of the <italic>CLSTM</italic> model is elucidated in model testing phase at study sites in Queensland, Australia. Using performance metrics, visual analysis of <italic>TSP</italic> simulations relative to observations, and detailed error analysis, this study ascertains the <italic>CLSTM</italic> model’s practical utility for air pollutant forecasting systems in health risk mitigation. This study captures a feasible opportunity to emulate air quality at relatively high temporal resolutions in global regions where air pollution is a considerable threat to public health.

Monitoring of environmental contaminants is a critical part of exposure sciences research and public health practice. Missing data are often encountered when performing short-term monitoring (<24 h) of air pollutants with real-time monitors, especially in resource-limited areas. Approaches for handling consecutive periods of missing and incomplete data in this context remain unclear. Our aim is to evaluate existing imputation methods for handling missing data for real-time monitors operating for short durations. In a current field-study, realtime PM2.5 monitors were placed outside of 20 households and ran for 24-hours. Missing data was simulated in these households at four consecutive periods of missingness (20%, 40%, 60%, 80%). Univariate (Mean, Median, Last Observation Carried Forward, Kalman Filter, Random, Markov) and multivariate time-series (Predictive Mean Matching, Row Mean Method) methods were used to impute missing concentrations, and performance was evaluated using five error metrics (Absolute Bias, Percent Absolute Error in Means, R2 Coefficient of Determination, Root Mean Square Error, Mean Absolute Error). Univariate methods of Markov, random, and mean imputations were the best performing methods that yielded 24-hour mean concentrations with the lowest error and highest R2 values across all levels of missingness. When evaluating error metrics minute-by-minute, Kalman filters, median, and Markov methods performed well at low levels of missingness (20-40%). However, at higher levels of missingness (60-80%), Markov, random, median, and mean imputation performed best on average. Multivariate methods were the worst performing imputation methods across all levels of missingness. Imputation using univariate methods may provide a reasonable solution to addressing missing data for short-term monitoring of air pollutants, especially in resource-limited areas. Further efforts are needed to evaluate imputation methods that are generalizable across a diverse range of study environments.

Modelling air quality with a practical tool that produces real-time forecasts to mitigate risk to public health continues to face significant challenges considering the chaotic, non-linear and high dimensional nature of air quality predictor variables. The novelty of this research is to propose a hybrid early-warning artificial intelligence (AI) framework that can emulate hourly air quality variables (i.e., Particulate Matter 2.5, PM2.5; Particulate Matter 10, PM10 and lower atmospheric visibility, VIS), the atmospheric variables associated with increased respiratory induced mortality and recurrent health-care cost. Firstly, hourly air quality data series (January-2015 to December-2017) are demarcated into their respective intrinsic mode functions (IMFs) and a residual sub-series that reveal patterns and resolve data complexity characteristics, followed by partial autocorrelation function applied to each IMF and residual sub-series to unveil historical changes in air quality. To design the prescribed hybrid model, the data is partitioned into training (70%), validation (15%) and testing (15%) sub-sets. The online sequential-extreme learning machine (OS-ELM) algorithm integrated with improved complete ensemble empirical mode decomposition with adaptive noise (ICEEMDAN) is designed as a data pre-processing system to robustly extract predictive patterns and fine-tune the model generalization to a near-optimal global solution, which represents modelled air quality at hourly forecast horizons. The resulting early warning AI-based framework denoted as ICEEMDAN-OS-ELM model, is individually constructed by forecasting each IMF and residual sub-series, with hourly PM2.5, PM10, and VIS obtained by the aggregated sum of forecasted IMFs and residual sub-series. The results are benchmarked with many competing predictive approaches; e.g., hybrid ICEEMDAN-multiple-linear regression (MLR), ICEEMDAN-M5 model tree and standalone versions: OS-ELM, MLR, M5 model tree. Statistical metrics including the root-mean-square error (RMSE), mean absolute error (MAE), Willmott's Index (WI), Legates & McCabe's Index (ELM) and Nash-Sutcliffe coefficients (ENS) are used to evaluate the model's accuracy. Both visual and statistical results show that the proposed ICEEMDAN-OS-ELM model registers superior results, outperforming alternative comparison approaches. For instance, for PM2.5,ELM values ranged from 0.65-0.82 vs. 0.59-0.77 for ICEEMDAN-M5 tree, 0.59-0.74 for ICEEMDAN-MLR, 0.28-0.54 for OS-ELM, 0.27-0.54 for M5 tree and 0.25-0.53 for the MLR model. For remaining air quality variables (i.e., PM10 & VIS), the objective model (ICEEMDAN-OS-ELM) outperformed the comparative models. In particular, ICEEMDAN-OS-ELM registered relatively low RMSE/MAE, ranging from approximately 0.7-1.03 μg/m3(MAE), 1.01-1.47 μg/m3(RMSE) for PM2.5 whereas for PM10, these metrics registered a value of 1.29-3.84 μg/m3(MAE), 3.01-7.04 μg/m3(RMSE) and for Visibility, they were 0.01-3.72 μg/m3 (MAE (Mm-1)), 0.04-5.98 μg/m3 (RMSE (Mm-1)). Visual analysis of forecasted and observed air quality through a Taylor diagram illustrates the objective model's preciseness, confirming the versatility of early warning AI-model in generating air quality forecasts. The excellent performance ascertains the hybrid model's potential utility for air quality monitoring and subsequent public health risk mitigation.

Missing values exist in many generated datasets in science. Therefore, utilizing missing data imputation methods is a common and important practice. These methods are a kind of treatment for uncertainty and vagueness existing in datasets. On the other hand, methods based on fuzzy-rough sets provide excellent tools for dealing with uncertainty, possessing highly desirable properties such as robustness and noise tolerance. Furthermore, they can find minimal representations of data and do not need potentially erroneous user inputs. As a result, utilizing fuzzy-rough sets for imputation should be an effective approach. In this paper, we propose three missing value imputation methods based on fuzzy-rough sets and its recent extensions; namely, implicator/t-norm based fuzzy-rough sets, vaguely quantified rough sets and also ordered weighted average based rough sets. These methods are compared against 11 state-of-the-art imputation methods implemented in the KEEL data mining software on 27 benchmark datasets. The results show, via non-parametric statistical analysis, that the proposed methods exhibit excellent performance in general. We have proposed 3 missing value imputation methods based on fuzzy-rough nearest neighbors: FRNNI, OWANNI and VQNNI.The results show they perform excellently.FRNNI outperforms the others.OWANNI is as good as FRNNI but in some cases FRNNI outperforms it.

Missing data has been a challenge in air quality measurement. In this study, we develop an input-adaptive proxy, which selects input variables of other air quality variables based on their correlation coefficients with the output variable. The proxy uses ordinary least squares regression model with robust optimization and limits the input variables to a maximum of three to avoid overfitting. The adaptive proxy learns from the data set and generates the best model evaluated by adjusted coefficient of determination (adjR2). In case of missing data in the input variables, the proposed adaptive proxy then uses the second-best model until all the missing data gaps are filled up. We estimated black carbon (BC) concentration by using the input-adaptive proxy in two sites in Helsinki, which respectively represent street canyon and urban background scenario, as a case study. Accumulation mode, traffic counts, nitrogen dioxide and lung deposited surface area are found as input variables in models with the top rank. In contrast to traditional proxy, which gives 20–80% of data, the input-adaptive proxy manages to give full continuous BC estimation. The newly developed adaptive proxy also gives generally accurate BC (street canyon: adjR2 = 0.86–0.94; urban background: adjR2 = 0.74–0.91) depending on different seasons and day of the week. Due to its flexibility and reliability, the adaptive proxy can be further extend to estimate other air quality parameters. It can also act as an air quality virtual sensor in support with on-site measurements in the future.

Missing data frequently happen in environmental research, usually due to faults in data acquisition, inadequate sampling or measurement error. They make difficult to determine whether the limits set by the European Community on certain indicators of air quality are fulfilled or not. Indeed, due to missing values, the number of exceedances per year of \(PM_{10}\), that is particulate matter 10 \(\upmu \)m or less in diameter, and other air quality indicators are often heavily underestimated, and no environmental policy is applied to protect citizen health.