Sensor Data Validation and Reconstruction

In a real water network, a telecontrol system must periodically acquire, store and validate sensor data to achieve accurate monitoring of the whole network in real time. For each sensor measurement, data are usually represented by one-dimensional time series. These values, known as raw data, need to be validated before further use to assure the reliability of the results obtained when using them. In real operation, problems affecting the communication system, lack of reliability of sensors or other inherent errors often arise, generating missing or false data during certain periods of time. These data must be detected and replaced by estimated data. Thus, it is important to provide the data system with procedures that can detect such problems and assist the user in monitoring and processing the incoming data. Data validation is an essential step to improve data reliability. The validated data represent measurements of the variables in the required form where unnecessary information from raw data has been removed. In this chapter, a methodology for data validation and reconstruction of sensor data collected from a water network is developed, taking into account not only spatial models, but also temporal models (time series of each sensor) and internal models of the several components in the local units (e.g., pumps, valves, flows, levels). The methodology is illustrated by means of its application to flow and level meters of the Catalonia Regional Water Network.

[1]  Magdalene Marinaki,et al.  Optimal Real-time Control of Sewer Networks , 2005 .

[2]  Gilles Mourot,et al.  Rainfall–runoff multi-modelling for sensor fault diagnosis , 2001 .

[3]  Vasilis Kanakoudis,et al.  Water pipe network reliability assessment using the DAC method , 2011 .

[4]  Steven X. Ding,et al.  Model-based Fault Diagnosis Techniques: Design Schemes, Algorithms, and Tools , 2008 .

[5]  Didier Graillot,et al.  Fault detection on a sewer network by a combination of a Kalman filter and a binary sequential probability ratio test , 2000 .

[6]  K M Tsang.  Sensor data validation using gray models. , 2003, ISA transactions.

[7]  Gilles Mourot,et al.  An improved PCA scheme for sensor FDI: Application to an air quality monitoring network , 2006 .

[8]  Gustaf Olsson,et al.  Instrumentation, Control and Automation in Wastewater Systems , 2015 .

[9]  V. Popov,et al.  Waste Management And the Environment III , 2006 .

[10]  Steven C. Wheelwright,et al.  Forecasting methods and applications. , 1979 .

[11]  J. W. Taylor,et al.  Short-term electricity demand forecasting using double seasonal exponential smoothing , 2003, J. Oper. Res. Soc..

[12]  Michèle Basseville,et al.  Detection of abrupt changes , 1993 .

[13]  Didier Maquin,et al.  Fault measurement detection in an urban water supply network , 2006 .

[14]  A. Ferrer,et al.  Dealing with missing data in MSPC: several methods, different interpretations, some examples , 2002 .

[15]  R. Brown.  Statistical forecasting for inventory control , 1960 .

[16]  Raghunathan Rengaswamy,et al.  A review of process fault detection and diagnosis: Part II: Qualitative models and search strategies , 2003, Comput. Chem. Eng..

[17]  Peter Steen Mikkelsen,et al.  Quality control of rain data used for urban runoff systems , 1997 .

[18]  Vicenç Puig,et al.  Inconsistent sensor data detection/correction: Application to environmental systems , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[19]  Saad Bennis,et al.  Improving single-variable and multivariable techniques for estimating missing hydrological data , 1997 .

[20]  M. Kramer.  Nonlinear principal component analysis using autoassociative neural networks , 1991 .

[21]  Christophe Croux,et al.  Robust Forecasting with Exponential and Holt-Winters Smoothing , 2007 .

[22]  Vasilis Kanakoudis,et al.  Pipe Networks Risk Assessment Based on Survival Analysis , 2011 .

[23]  P A Vanrolleghem,et al.  Sensor validation and reconciliation for a partial nitrification process. , 2006, Water science and technology : a journal of the International Association on Water Pollution Research.

[24]  P. A. Taylor,et al.  Missing data methods in PCA and PLS: Score calculations with incomplete observations , 1996 .

[25]  Vicenç Puig,et al.  Combining contaminant event diagnosis with data validation/reconstruction: Application to smart buildings , 2014, 22nd Mediterranean Conference on Control and Automation.

[26]  M. C. Deo,et al.  Filling up gaps in wave data with genetic programming , 2008 .

[27]  Vicenç Puig,et al.  Sensor Data Validation and Reconstruction in Water Networks: A Methodology and Software Implementation , 2014, CRITIS.

[28]  S. Ciavatta,et al.  The Extended Kalman Filter (EKF) as a tool for the assimilation of high frequency water quality data , 2003 .

[29]  M Mourad,et al.  A method for automatic validation of long time series of data in urban hydrology. , 2002, Water science and technology : a journal of the International Association on Water Pollution Research.

[30]  M. B. Beck,et al.  Fault detection in a real-time monitoring network for water quality in the lagoon of Venice (Italy). , 2004, Water science and technology : a journal of the International Association on Water Pollution Research.

[31]  D Burnell.  Auto-validation of district meter data , 2003 .

[32]  Saad Bennis,et al.  Multivariate Technique for Validating Historical Hydrometric Data with Redundant Measurements , 2000 .

[33]  Zoran Kapelan,et al.  Improved real-time data anomaly detection using context classification , 2011 .

[34]  G Lobanova,et al.  Approach for restoration of missing data, long-term time series and generalization of results , 2003 .

[35]  Vicenç Puig,et al.  Fault Detection and Isolation of Rain Gauges and Limnimeters of Barcelona's Sewer System Using Interval Models , 2003 .

[36]  James W. Taylor,et al.  Triple seasonal methods for short-term electricity demand forecasting , 2010, Eur. J. Oper. Res..

[37]  P. Vanrolleghem,et al.  Real time control of urban wastewater systems: where do we stand today? , 2004 .

[38]  Vicenç Puig,et al.  Validation and reconstruction of flow meter data in the Barcelona water distribution network , 2010 .

[39]  E. S. Gardner.  EXPONENTIAL SMOOTHING: THE STATE OF THE ART, PART II , 2006 .

[40]  Thierry Denoeux,et al.  A neural network-based software sensor for coagulation control in a water treatment plant , 2001, Intell. Data Anal..

[41]  V. K. Kanakoudis,et al.  The role of leaks and breaks in water networks: technical and economical solutions , 2001 .

[42]  Santiago Espin,et al.  Methodology of a data validation and reconstructions tool to improve the realiability of the water network supervision , 2010 .

[43]  É. Crobeddu,et al.  Data Acquisition, Validation And Forecasting ForA Combined Sewer Network , 2006 .

[44]  Slobodan P. Simonovic,et al.  Estimation of missing streamflow data using principles of chaos theory , 2002 .

[45]  Garth P. McCormick,et al.  Communications to the Editor--Exponential Forecasting: Some New Variations , 1969 .

[46]  Vicenç Puig,et al.  Fault diagnosis and fault tolerant control using set-membership approaches: Application to real case studies , 2010, Int. J. Appl. Math. Comput. Sci..

[47]  B. Maul-Kötter,et al.  Correction and preparation of continuously measured raingauge data : A standard method in North Rhine-Westphalia , 1998 .

[48]  Gwilym M. Jenkins,et al.  Time series analysis, forecasting and control , 1972 .

[49]  Peter R. Winters,et al.  Forecasting Sales by Exponentially Weighted Moving Averages , 1960 .