Scalable SUM-Shrinkage Schemes for Distributed Monitoring Large-Scale Data Streams

In this article, motivated by biosurveillance and censoring sensor networks, we investigate the problem of distributed monitoring large-scale data streams where an undesired event may occur at some unknown time and affect only a few unknown data streams. We propose to develop scalable global monitoring schemes by parallel running local detection procedures and by combining these local procedures together to make a global decision based on SUM-shrinkage techniques. Our approach is illustrated in two concrete examples: one is the nonhomogeneous case when the pre-change and post-change local distributions are given, and the other is the homogeneous case of monitoring a large number of independent $N(0,1)$ data streams where the means of some data streams might shift to unknown positive or negative values. Numerical simulation studies demonstrate the usefulness of the proposed schemes.

[1]  Yajun Mei,et al.  Large-Scale Multi-Stream Quickest Change Detection via Shrinkage Post-Change Estimation , 2015, IEEE Transactions on Information Theory.

[2]  Yajun Mei,et al.  Quickest Change Detection and Kullback-Leibler Divergence for Two-State Hidden Markov Models , 2015, IEEE Transactions on Signal Processing.

[3]  Moshe Pollak,et al.  Sequential Change-Point Detection Procedures That are Nearly Optimal and Computationally Simple , 2008 .

[4]  Yajun Mei,et al.  Quickest detection in censoring sensor networks , 2011, 2011 IEEE International Symposium on Information Theory Proceedings.

[5]  Taposh Banerjee,et al.  Data-Efficient Quickest Change Detection in Sensor Networks , 2015, IEEE Transactions on Signal Processing.

[6]  Michèle Basseville,et al.  Detection of abrupt changes: theory and application , 1993 .

[7]  M. Basseville,et al.  Sequential Analysis: Hypothesis Testing and Changepoint Detection , 2014 .

[8]  D. Siegmund.  Sequential Analysis: Tests and Confidence Intervals , 1985 .

[9]  Yongguo Mei,et al.  Information bounds and quickest change detection in decentralized decision systems , 2005, IEEE Transactions on Information Theory.

[10]  C'eline L'evy-Leduc,et al.  Detection and localization of change-points in high-dimensional network traffic data , 2009, 0908.2310.

[11]  Martin Kulldorff,et al.  Prospective time periodic geographical disease surveillance using a scan statistic , 2001 .

[12]  I. Johnstone,et al.  Ideal spatial adaptation by wavelet shrinkage , 1994 .

[13]  A. Shiryaev.  On Optimum Methods in Quickest Detection Problems , 1963 .

[14]  Emmanuel J. Cand.  Modern statistical estimation via oracle inequalities , 2006 .

[15]  M. Pollak.  Optimal Detection of a Change in Distribution , 1985 .

[16]  E. S. Page.  CONTINUOUS INSPECTION SCHEMES , 1954 .

[17]  Y. Mei.  Efficient scalable schemes for monitoring a large number of data streams , 2010 .

[18]  M. Pollak.  Average Run Lengths of an Optimal Method of Detecting a Change in Distribution. , 1987 .

[19]  Douglas L. Jones,et al.  Energy-efficient detection in sensor networks , 2005, IEEE Journal on Selected Areas in Communications.

[20]  Moe Z. Win,et al.  Asymptotic Performance of a Censoring Sensor Network , 2007, IEEE Transactions on Information Theory.

[21]  R. Durrett.  Probability: Theory and Examples , 1993 .

[22]  G. Lorden.  PROCEDURES FOR REACTING TO A CHANGE IN DISTRIBUTION , 1971 .

[23]  Rebecca Willett,et al.  Change-Point Detection for High-Dimensional Time Series With Missing Data , 2012, IEEE Journal of Selected Topics in Signal Processing.

[24]  Douglas C. Montgomery,et al.  Introduction to Statistical Quality Control , 1986 .

[25]  Jianqing Fan,et al.  Test of Significance When Data Are Curves , 1998 .

[26]  W. Shewhart.  The Economic Control of Quality of Manufactured Product. , 1932 .

[27]  T. Lai.  SEQUENTIAL ANALYSIS: SOME CLASSICAL PROBLEMS AND NEW CHALLENGES , 2001 .

[28]  J. Naus,et al.  Scan Statistics , 2014, Encyclopedia of Social Network Analysis and Mining.

[29]  S. W. Roberts.  A Comparison of Some Control Chart Procedures , 1966 .

[30]  P. D. T. O'Connor.  Introduction to Statistical Quality Control (2nd edition), D. C. Montgomery, Wiley, 1991. Number of pages: 702. £49.35, Paperback £17.50 , 1991 .

[31]  J. Neyman.  »Smooth test» for goodness of fit , 1937 .

[32]  Leo Breiman,et al.  Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author) , 2001, Statistical Science.

[33]  Yajun Mei,et al.  An Adaptive Sampling Strategy for Online High-Dimensional Process Monitoring , 2015, Technometrics.

[34]  Michèle Basseville,et al.  Detection of Abrupt Changes: Theory and Applications. , 1995 .

[35]  J. Kiefer,et al.  Asymptotically Optimum Sequential Inference and Design , 1963 .

[36]  D. Siegmund,et al.  Sequential multi-sensor change-point detection , 2012, 2013 Information Theory and Applications Workshop (ITA).

[37]  G. Moustakides.  Optimal stopping times for detecting changes in distributions , 1986 .

[38]  L. Gordon,et al.  An Efficient Sequential Nonparametric Scheme for Detecting a Change of Distribution , 1994 .

[39]  Venugopal V. Veeravalli.  Decentralized quickest change detection , 2001, IEEE Trans. Inf. Theory.

[40]  Y. Bar-Shalom,et al.  Censoring sensors: a low-communication-rate scheme for distributed detection , 1996, IEEE Transactions on Aerospace and Electronic Systems.

[41]  Rudolf B. Blazek,et al.  Detection of intrusions in information systems by sequential change-point methods , 2005 .