Automated detection of unusual soil moisture probe response patterns with association rule learning

Abstract In-situ field monitoring networks generate vast quantities of continuous data can help to improve the design, management, operation and maintenance of Green Infrastructure (GI) systems. However, such actions require efficient and reliable quality assurance quality control (QAQC). In this paper, we develop a rule-based learning algorithm involving Dynamic Time Warping (DTW) to investigate the feasibility of detecting anomalous responses from soil moisture probes using data collected from a GI site in Milwaukee, WI. As an enhancement to traditional QAQC methods which rely on individual time steps, this method converts the continuous time series into event sequences from which response patterns can be detected. Association rules are developed on both environmental features and event features. The results suggest that this method could be used to identify abnormal change patterns as compared to intra-site historical observations. Though developed for soil moisture, this method could easily be extended to apply on other continuous environmental datasets.

[1]  Russell S. Vose,et al.  Comprehensive Automated Quality Assurance of Daily Surface Observations , 2010 .

[2]  S. Gaffin,et al.  Green Roof Hydrology: Results from a Small-Scale Lysimeter Setup (Bronx, NY) , 2010 .

[3]  Lei Chen,et al.  In-network Outlier Cleaning for Data Collection in Sensor Networks , 2006, CleanDB.

[4]  David R. Easterling,et al.  Quality Control of Pre-1948 Cooperative Observer Network Data , 2005 .

[5]  Antonino Cancelliere,et al.  Quality control of daily rainfall data with neural networks , 2009 .

[6]  George H. John.  Robust Decision Trees: Removing Outliers from Databases , 1995, KDD.

[7]  Hui Lin,et al.  Application of Association Rule in Disaster Weather Forecasting , 2004, Ann. GIS.

[8]  Eamonn Keogh.  Exact Indexing of Dynamic Time Warping , 2002, VLDB.

[9]  Eyal Amir,et al.  Real‐time Bayesian anomaly detection in streaming environmental data , 2007 .

[10]  Virginia Stovin,et al.  Moisture content behaviour in extensive green roofs during dry periods: The influence of vegetation and substrate characteristics , 2014 .

[11]  Donald J. Berndt,et al.  Using Dynamic Time Warping to Find Patterns in Time Series , 1994, KDD Workshop.

[12]  Daniel E. Marasco,et al.  Hydrological performance of extensive green roofs in New York City: observations and multi-year modeling of three full-scale systems , 2013 .

[13]  Ramakrishnan Srikant,et al.  Fast algorithms for mining association rules , 1998, VLDB 1998.

[14]  George Tzanetakis,et al.  Polyphonic audio matching and alignment for music retrieval , 2003, 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (IEEE Cat. No.03TH8684).

[15]  D. Hand,et al.  Unsupervised Profiling Methods for Fraud Detection , 2002 .

[16]  Ambuj K. Singh,et al.  Real-time nondestructive structural health monitoring using support vector machines and wavelets , 2005, SPIE Smart Structures and Materials + Nondestructive Evaluation and Health Monitoring.

[17]  Peter Arzberger,et al.  New Eyes on the World: Advanced Sensors for Ecology , 2009 .

[18]  Juan Vicente Giráldez,et al.  A new quality control procedure based on non-linear autoregressive neural network for validating raw river stage data , 2014 .

[19]  Jinsheng You,et al.  Sensitivity Analysis of Quality Assurance Using the Spatial Regression Approach—A Case Study of the Maximum/Minimum Air Temperature , 2005 .