Unsupervised detection of contextual anomaly in remotely sensed data

Abstract Massive amounts of remotely sensed data are being generated at an unprecedented rate, offering new opportunities for data-driven scientific discovery in the Earth sciences and related domains. However, due to the sheer volume of remotely sensed data and the lack of effective data analytics tools, most data remain in the dark, with little to no quality assurance and limited access to high-level analytical tools. Anomaly detection, which aims to find scenarios that differ from the norm, is of particular importance when analyzing remotely sensed data. However, most previous work has focused on identifying individual anomalies, and required prior knowledge of the ground truth for supervised learning. In this work, we propose an unsupervised anomaly detection framework that requires no prior knowledge and is capable of detecting anomalous events, which we define as groups of outlier objects differing contextually from their spatial and temporal neighbors. Such contextual anomalies can be useful in discovering both hidden quality issues in the data and real natural events of significance. We demonstrate the effectiveness of our framework via Web-based tools developed for visualizing and analyzing such contextual anomalies, using two types of data. The techniques and tools developed in this project are generally usable for a diverse set of satellite products and will be made publicly available with the support of the National Snow and Ice Data Center (NSIDC).

[1]  Julia Boike,et al.  Spatio-temporal sensitivity of MODIS land surface temperature anomalies indicates high potential for large-scale land cover change detection in Arctic permafrost landscapes , 2014 .

[2]  Albert Y. Zomaya,et al.  Remote sensing big data computing: Challenges and opportunities , 2015, Future Gener. Comput. Syst..

[3]  Klaus Dethloff,et al.  Corrigendum: Recent changes in Arctic temperature extremes: warm and cold spells during winter and summer (2015 Environ. Res. Lett. 10 114020) , 2016 .

[4]  Raymond T. Ng,et al.  Distance-based outliers: algorithms and applications , 2000, The VLDB Journal.

[5]  Raymond T. Ng,et al.  Algorithms for Mining Distance-Based Outliers in Large Datasets , 1998, VLDB.

[6]  Gabriele Villarini,et al.  Detecting inhomogeneities in the Twentieth Century Reanalysis over the central United States , 2012 .

[7]  Zhilin Li,et al.  A Multiscale Approach for Spatio‐Temporal Outlier Detection , 2006, Trans. GIS.

[8]  Sanjay Ranka,et al.  Conditional Anomaly Detection , 2007, IEEE Transactions on Knowledge and Data Engineering.

[9]  Dongmei Chen,et al.  Change detection from remotely sensed images: From pixel-based to object-based approaches , 2013 .

[10]  Awais Ahmad,et al.  Real-Time Big Data Analytical Architecture for Remote Sensing Application , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[11]  Volker Roth,et al.  Outlier Detection with One-class Kernel Fisher Discriminants , 2004, NIPS.

[12]  Stefan Rahmstorf,et al.  A decade of weather extremes , 2012 .

[13]  Son V. Nghiem,et al.  The extreme melt across the Greenland ice sheet in 2012 , 2012 .

[14]  Erik Borg,et al.  Assessment for Remote Sensing Data: Accuracy of Interactive Data Quality Interpretation , 2011, ICCSA.

[15]  Arun Kejariwal,et al.  A Novel Technique for Long-Term Anomaly Detection in the Cloud , 2014, HotCloud.

[16]  Bin Jiang,et al.  Geospatial Big Data Handling Theory and Methods: A Review and Research Challenges , 2015, ArXiv.

[17]  Son V. Nghiem,et al.  The melt anomaly of 2002 on the Greenland Ice Sheet from active and passive microwave satellite observations , 2004 .

[18]  P. Rousseeuw.  Silhouettes: a graphical aid to the interpretation and validation of cluster analysis , 1987 .

[19]  N. Giri.  Multivariate Statistical Analysis : Revised And Expanded , 2003 .

[20]  Soille Pierre,et al.  ESA-EUSC-JRC 2011 - The proceedings of the Seventh Conference on Image Information Mining: Geospatial Intelligence from Earth Observation , 2011 .

[21]  Terry V. Callaghan,et al.  Vegetation recovery following extreme winter warming events in the sub-Arctic estimated using NDVI from remote sensing and handheld passive proximal sensors , 2012 .

[22]  Gyemin Lee,et al.  EM algorithms for multivariate Gaussian mixture models with truncated and censored data , 2012, Comput. Stat. Data Anal..

[23]  Sanjay Chawla,et al.  On local spatial outliers , 2004, Fourth IEEE International Conference on Data Mining (ICDM'04).

[24]  G. Meehl,et al.  Climate extremes: observations, modeling, and impacts. , 2000, Science.

[25]  Christopher S. Lynnes,et al.  Automated Data Quality Assessment in the Intelligent Archive , 2003 .

[26]  Derya Birant,et al.  ST-DBSCAN: An algorithm for clustering spatial-temporal data , 2007, Data Knowl. Eng..

[27]  Charu C. Aggarwal,et al.  Outlier Detection for Temporal Data , 2014, Outlier Detection for Temporal Data.

[28]  Kanishka Bhaduri,et al.  Algorithms for speeding up distance-based outlier detection , 2011, KDD.

[29]  M Stein,et al.  North Atlantic Subpolar Gyre Warming - Impacts on Greenland Offshore Waters , 2005 .

[30]  Sergio M. Vicente-Serrano,et al.  Observed trends and future projections for winter warm events in the Ebro basin, northeast Iberian Peninsula , 2014 .

[31]  Jeff G. Schneider,et al.  Anomaly pattern detection in categorical datasets , 2008, KDD.

[32]  Alexander Barth,et al.  Outlier detection in satellite data using spatial coherence , 2012 .

[33]  Barnabás Póczos,et al.  Group Anomaly Detection using Flexible Genre Models , 2011, NIPS.

[34]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[35]  Klaus Dethloff,et al.  Recent changes in Arctic temperature extremes: warm and cold spells during winter and summer , 2015 .

[36]  Aaron M. McCright,et al.  The impacts of temperature anomalies and political orientation on perceived winter warming , 2014 .