Anomaly Detection in Streaming Sensor Data

In this chapter we consider a cell phone network as a set of automatically deployed sensors that records movement and interaction patterns of the population. We discuss methods for detecting anomalies in the streaming data produced by the cell phone network. We motivate this discussion by describing the Wireless Phone Based Emergency Response (WIPER) system, a proof-of-concept decision support system for emergency response managers. We also discuss some of the scientific work enabled by this type of sensor data and the related privacy issues. We describe scientific studies that use the cell phone data set and steps we have taken to ensure the security of the data. We describe the overall decision support system and discuss three methods of anomaly detection that we have applied to the data.

[1]  Massimo Marchiori,et al.  Error and attacktolerance of complex network s , 2004 .

[2]  Mark E. J. Newman,et al.  Structure and Dynamics of Networks , 2009 .

[3]  Corinna Cortes,et al.  Computational Methods for Dynamic Graphs , 2003 .

[4]  G. Madey,et al.  Anomaly Detection in the WIPER System Using Markov Modulated Poisson Process , 2007 .

[5]  Albert-László Barabási,et al.  Error and attack tolerance of complex networks , 2000, Nature.

[6]  W. R. Buckland,et al.  Outliers in Statistical Data , 1979 .

[7]  Albert-László Barabási,et al.  WIPER: A Multi-Agent System for Emergency Response , 2006 .

[8]  James H. Moor,et al.  Towards a theory of privacy in the information age , 1997, CSOC.

[9]  Benjamin Gerber,et al.  Conceptualizing privacy , 2010, CSOC.

[10]  Yehuda Lindell,et al.  Privacy Preserving Data Mining , 2002, Journal of Cryptology.

[11]  Philip S. Yu,et al.  A Framework for Clustering Evolving Data Streams , 2003, VLDB.

[12]  Mark E. J. Newman,et al.  The Structure and Function of Complex Networks , 2003, SIAM Rev..

[13]  Duncan J. Watts,et al.  Collective dynamics of ‘small-world’ networks , 1998, Nature.

[14]  A. C. Evans.  European data protection law , 1981 .

[15]  Ping Yan,et al.  Anomaly Detection in Streaming Sensor Data , 2008 .

[16]  Carla E. Brodley,et al.  Feature Selection for Unsupervised Learning , 2004, J. Mach. Learn. Res..

[17]  Albert,et al.  Emergence of scaling in random networks , 1999, Science.

[18]  M. Newman.  Clustering and preferential attachment in growing networks. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[19]  Eli Upfal,et al.  Probability and Computing: Randomized Algorithms and Probabilistic Analysis , 2005 .

[20]  Sean D. Murphy.  U. S. EU “Safe Harbor” Data Privacy Arrangement , 2001, American Journal of International Law.

[21]  Padhraic Smyth,et al.  Learning to detect events with Markov-modulated poisson processes , 2007, TKDD.

[22]  C. A. Murthy,et al.  Unsupervised Feature Selection Using Feature Similarity , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Diane J. Cook,et al.  Graph-based anomaly detection , 2003, KDD '03.

[24]  Sudipto Guha,et al.  Clustering Data Streams: Theory and Practice , 2003, IEEE Trans. Knowl. Data Eng..

[25]  Alec Pawling,et al.  WIPER : An Emergency Response System , 2008 .

[26]  S. Redner,et al.  Introduction To Percolation Theory , 2018 .

[27]  Lawrence B. Holder,et al.  Structure Discovery in Sequentially-connected Data Streams , 2006, Int. J. Artif. Intell. Tools.

[28]  Victoria J. Hodge,et al.  A Survey of Outlier Detection Methodologies , 2004, Artificial Intelligence Review.

[29]  Jack Dongarra,et al.  Proceedings of the 9th International Conference on Computational Science , 2009, ICCS 2009.

[30]  Robert Tibshirani,et al.  Hybrid hierarchical clustering with applications to microarray data. , 2005, Biostatistics.

[31]  Lawrence B. Holder,et al.  Substructure Discovery Using Minimum Description Length and Background Knowledge , 1993, J. Artif. Intell. Res..

[32]  Bogdan Tatomir.  Crisis Management using Mobile ad-hoc Wireless Networks , 2005 .

[33]  Lise Getoor,et al.  Link mining: a survey , 2005, SKDD.

[34]  Marc P. Armstrong,et al.  Geographic Information Technologies and Personal Privacy , 2005, Cartogr. Int. J. Geogr. Inf. Geovisualization.

[35]  G. Madey,et al.  Uncovering individual and collective human dynamics from mobile phone records , 2007, 0710.2939.

[36]  Vincent Kanade,et al.  Clustering Algorithms , 2021, Wireless RF Energy Transfer in the Massive IoT Era.

[37]  A. Barabasi,et al.  Analysis of a large-scale weighted network of one-to-one human communication , 2007, physics/0702158.

[38]  Douglas H. Fisher,et al.  Knowledge Acquisition Via Incremental Conceptual Clustering , 1987, Machine Learning.

[39]  Chris Clifton,et al.  Defining Privacy for Data Mining , 2002 .

[40]  Manoj A. Thomas,et al.  EVResponse - Moving Beyond Traditional Emergency Response Notification , 2005, AMCIS.

[41]  Mark Newman,et al.  Detecting community structure in networks , 2004 .

[42]  Daniel J. Solove,et al.  'I've Got Nothing to Hide' and Other Misunderstandings of Privacy , 2007 .

[43]  Gregory R. Madey,et al.  Evaluation of Measurement Techniques for the Validation of Agent-Based Simulations Against Streaming Data , 2008, ICCS.

[44]  Tian Zhang,et al.  BIRCH: an efficient data clustering method for very large databases , 1996, SIGMOD '96.

[45]  M E J Newman.  Assortative mixing in networks. , 2002, Physical review letters.

[46]  Carson C. Chow,et al.  Small Worlds , 2000 .

[47]  Nitesh V. Chawla,et al.  Anomaly detection in a mobile communication network , 2007, Comput. Math. Organ. Theory.

[48]  Nitesh V. Chawla,et al.  Enhanced Situational Awareness: Application of DDDAS Concepts to Emergency and Disaster Management , 2007, International Conference on Computational Science.

[49]  Douglas R. Stinson,et al.  Cryptography: Theory and Practice , 1995 .

[50]  Chee Keong Kwoh,et al.  On the Two-level Hybrid Clustering Algorithm , 2004 .

[51]  Albert-László Barabási,et al.  WIPER: The Integrated Wireless Phone Based Emergency Response System , 2006, International Conference on Computational Science.

[52]  Greg Madey,et al.  WIPER: Leveraging the Cell Phone Network for Emergency Response † , 2006 .

[53]  Joao Antonio Pereira,et al.  Linked: The new science of networks , 2002 .

[54]  A-L Barabási,et al.  Structure and tie strengths in mobile communication networks , 2006, Proceedings of the National Academy of Sciences.

[55]  Josh Benaloh,et al.  Dense Probabilistic Encryption , 1999 .

[56]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[57]  Padhraic Smyth,et al.  Adaptive event detection with time-varying poisson processes , 2006, KDD '06.

[58]  Ran Wolff,et al.  k-Anonymous Decision Tree Induction , 2006, PKDD.

[59]  Mihai Surdeanu,et al.  A hybrid unsupervised approach for document clustering , 2005, KDD '05.

[60]  Anil K. Jain,et al.  Data clustering: a review , 1999, CSUR.

[61]  Albert-László Barabási,et al.  Statistical mechanics of complex networks , 2001, ArXiv.

[62]  Pascal Paillier,et al.  Public-Key Cryptosystems Based on Composite Degree Residuosity Classes , 1999, EUROCRYPT.

[63]  Rajeev Motwani,et al.  Incremental clustering and dynamic information retrieval , 1997, STOC '97.

[64]  Chen Sun,et al.  Handbook on Advancements in Smart Antenna Technologies for Wireless Networks , 2008 .

[65]  L Sweeney,et al.  Weaving Technology and Policy Together to Maintain Confidentiality , 1997, Journal of Law, Medicine & Ethics.

[66]  Nilson Arrais.  Quality control handbook , 1966 .

[67]  Chris Clifton,et al.  When do data mining results violate privacy? , 2004, KDD.