A Hybrid Approach for Alarm Verification using Stream Processing, Machine Learning and Text Analytics

False alarms triggered by security sensors incur high costs for all parties involved. According to police reports, a large majority of alarms are false. Recent advances in machine learning can enable automatically classifying alarms. However, building a scalable alarm verification system is a challenge, since the system needs to: (1) process thousands of alarms in real-time, (2) classify false alarms with high accuracy and (3) perform historic data analysis to enable better insights into the results for human operators. This requires a mix of machine learning, stream and batch processing – technologies which are typically optimized independently. We combine all three into a single, real-world application. This paper describes the implementation and evaluation of an alarm verification system we developed jointly with Sitasys, the market leader in alarm transmission in central Europe. Our system can process around 30K alarms per second with a verification accuracy of above 90%.

[1]  Seif Haridi,et al.  Apache Flink™: Stream and Batch Processing in a Single Engine , 2015, IEEE Data Eng. Bull..

[2]  Nesime Tatbul,et al.  DejaVu: declarative pattern matching over live and archived streams of events , 2009, SIGMOD Conference.

[3]  Yuh-Jye Lee,et al.  Semi-supervised Learning for False Alarm Reduction , 2010, ICDM.

[4]  Terran Lane,et al.  An Application of Machine Learning to Anomaly Detection , 1999 .

[5]  Surajit Chaudhuri,et al.  An overview of data warehousing and OLAP technology , 1997, SGMD.

[6]  Qiang Chen,et al.  Aurora : a new model and architecture for data stream management ) , 2006 .

[7]  Arie Shoshani,et al.  Enabling Real-Time Querying of Live and Historical Stream Data , 2007, 19th International Conference on Scientific and Statistical Database Management (SSDBM 2007).

[8]  Nathan Marz,et al.  Big Data: Principles and best practices of scalable realtime data systems , 2015 .

[9]  Hideki Imai,et al.  IDS False Alarm Reduction Using Continuous and Discontinuous Patterns , 2005, ACNS.

[10]  Lam-For Kwok,et al.  Adaptive False Alarm Filter Using Machine Learning in Intrusion Detection , 2011 .

[11]  Kurt Stockinger,et al.  Applied Data Science: Using Machine Learning for Alarm Verification , 2016, ERCIM News.

[12]  Prashant J. Shenoy,et al.  SCALLA: A Platform for Scalable One-Pass Analytics Using MapReduce , 2012, TODS.

[13]  Vincenza Carchiolo,et al.  An Intelligent and Pervasive Surveillance System for Home Security , 2012, Int. J. Comput. Commun. Control.

[14]  Lam For Kwok,et al.  IDS False Alarm Filtering Using KNN Classifier , 2004, WISA.

[15]  Mourad Khayati,et al.  Online Anomaly Detection over Big Data Streams , 2019, Applied Data Science.

[16]  Jeffrey Davis,et al.  Continuous analytics over discontinuous streams , 2010, SIGMOD Conference.

[17]  Michael Stonebraker,et al.  Monitoring Streams - A New Class of Data Management Applications , 2002, VLDB.