A Comparative Study of Time Series Anomaly Detection Models for Industrial Control Systems

Anomaly detection has been known as an effective technique to detect faults or cyber-attacks in industrial control systems (ICS). Therefore, many anomaly detection models have been proposed for ICS. However, most models have been implemented and evaluated under specific circumstances, which leads to confusion about choosing the best model in a real-world situation. In other words, there still needs to be a comprehensive comparison of state-of-the-art anomaly detection models with common experimental configurations. To address this problem, we conduct a comparative study of five representative time series anomaly detection models: InterFusion, RANSynCoder, GDN, LSTM-ED, and USAD. We specifically compare the performance analysis of the models in detection accuracy, training, and testing times with two publicly available datasets: SWaT and HAI. The experimental results show that the best model results are inconsistent with the datasets. For SWaT, InterFusion achieves the highest F1-score of 90.7% while RANSynCoder achieves the highest F1-score of 82.9% for HAI. We also investigate the effects of the training set size on the performance of anomaly detection models. We found that about 40% of the entire training set would be sufficient to build a model producing a similar performance compared to using the entire training set.

[1]  S. Shitharth,et al.  Optimal Feature Selection Based on Evolutionary Algorithm for Intrusion Detection , 2022, SN Computer Science.

[2]  Ziwen Cai,et al.  Compliance Checking Based Detection of Insider Threat in Industrial Control System of Power Utilities , 2022, 2022 7th Asia Conference on Power and Electrical Engineering (ACPEE).

[3]  Muhammad Taha Jilani,et al.  Machine Learning for Intrusion Detection in Industrial Control Systems: Applications, Challenges, and Recommendations , 2022, Int. J. Crit. Infrastructure Prot..

[4]  Zivana Jakovljevic,et al.  CNN based method for the development of cyber-attacks detection algorithms in industrial control systems , 2021, Comput. Secur..

[5]  Ali A. Ghorbani,et al.  An evaluation framework for industrial control system cyber incidents , 2021, Int. J. Crit. Infrastructure Prot..

[6]  Pete Burnap,et al.  Cybersecurity of Industrial Cyber-Physical Systems: A Review , 2021, ACM Comput. Surv..

[7]  Jose A. Lozano,et al.  A Review on Outlier/Anomaly Detection in Time Series Data , 2020, ACM Comput. Surv..

[8]  Xiong Luo,et al.  Abnormal detection technology of industrial control system based on transfer learning , 2022, Appl. Math. Comput..

[9]  Dan Pei,et al.  Multivariate Time Series Anomaly Detection and Interpretation using Hierarchical Inter-Metric and Temporal Embedding , 2021, KDD.

[10]  Tomer Lancewicki,et al.  Practical Approach to Asynchronous Multivariate Time Series Anomaly Detection and Localization , 2021, KDD.

[11]  Jeong-Han Yun,et al.  Two ICS Security Datasets and Anomaly Detection Contest on the HIL-based Augmented ICS Testbed , 2021, CSET @ USENIX Security Symposium.

[12]  Bryan Hooi,et al.  Graph Neural Network-Based Anomaly Detection in Multivariate Time Series , 2021, AAAI.

[13]  Aditya P. Mathur,et al.  A multilayer perceptron model for anomaly detection in water treatment plants , 2020, Int. J. Crit. Infrastructure Prot..

[14]  Jörg Franke,et al.  An Analysis of Black Energy 3, Crashoverride, and Trisis, Three Malware Approaches Targeting Operational Technology Systems , 2020, 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA).

[15]  Maria A. Zuluaga,et al.  USAD: UnSupervised Anomaly Detection on Multivariate Time Series , 2020, KDD.

[16]  Mauro Conti,et al.  Sorry, Shodan is not Enough! Assessing ICS Security via IXP Network Traffic Analysis , 2020, ArXiv.

[17]  Marcin Nawrocki,et al.  Uncovering Vulnerable Industrial Control Systems from the Internet Core , 2019, NOMS 2020 - 2020 IEEE/IFIP Network Operations and Management Symposium.

[18]  Wei Sun,et al.  Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network , 2019, KDD.

[19]  Abbas Jamalipour,et al.  Protecting Cyber Physical Systems Using a Learned MAPE-K Model , 2019, IEEE Access.

[20]  G. Russell,et al.  WaterLeakage: A Stealthy Malware for Data Exfiltration on Industrial Control Systems Using Visual Channels* , 2019, 2019 IEEE 15th International Conference on Control and Automation (ICCA).

[21]  Taghi M. Khoshgoftaar,et al.  Survey on deep learning with class imbalance , 2019, J. Big Data.

[22]  Lei Shi,et al.  MAD-GAN: Multivariate Anomaly Detection for Time Series Data with Generative Adversarial Networks , 2019, ICANN.

[23]  Jamie B. Coble,et al.  Multilayer Data-Driven Cyber-Attack Detection System for Industrial Control Systems Based on Network, System, and Process Data , 2019, IEEE Transactions on Industrial Informatics.

[24]  Bo Zong,et al.  A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data , 2018, AAAI.

[25]  Chunming Wu,et al.  Enhanced Cyber-Physical Security through Deep Learning Techniques , 2019, CPS Summer School, PhD Workshop.

[26]  Keisuke Ishibashi,et al.  Estimation of Dimensions Contributing to Detected Anomalies with Variational Autoencoders , 2018, ArXiv.

[27]  Henrik Sandberg,et al.  A Survey of Physics-Based Attack Detection in Cyber-Physical Systems , 2018, ACM Comput. Surv..

[28]  Naghmeh Moradpoor,et al.  A supervised energy monitoring-based machine learning approach for anomaly detection in a clean water supply system , 2018, 2018 International Conference on Cyber Security and Protection of Digital Services (Cyber Security).

[29]  Weisong Shi,et al.  On security challenges and open issues in Internet of Things , 2018, Future Gener. Comput. Syst..

[30]  Jianying Zhou,et al.  NoisePrint: Attack Detection Using Sensor and Process Noise Fingerprint in Cyber Physical Systems , 2018, AsiaCCS.

[31]  Bo Zong,et al.  Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection , 2018, ICLR.

[32]  Valentino Constantinou,et al.  Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding , 2018, KDD.

[33]  Jian Fu,et al.  A Novel Data Analytical Approach for False Data Injection Cyber-Physical Attack Mitigation in Smart Grids , 2017, IEEE Access.

[34]  Jun Sun,et al.  Anomaly Detection for a Water Treatment System Using Unsupervised Machine Learning , 2017, 2017 IEEE International Conference on Data Mining Workshops (ICDMW).

[35]  D. Prince Winston,et al.  An enhanced optimization based algorithm for intrusion detection in SCADA network , 2017, Comput. Secur..

[36]  Osama A. Mohammed,et al.  A Survey on Smart Grid Cyber-Physical System Testbeds , 2017, IEEE Communications Surveys & Tutorials.

[37]  Kaspersky Lab,et al.  THREAT LANDSCAPE FOR INDUSTRIAL AUTOMATION SYSTEMS IN THE SECOND HALF OF 2016 , 2017 .

[38]  Song Tan,et al.  Survey of Security Advances in Smart Grid: A Data Driven Approach , 2017, IEEE Communications Surveys & Tutorials.

[39]  Haibo He,et al.  Cyber-physical attacks and defences in the smart grid: a survey , 2016, IET Cyper-Phys. Syst.: Theory & Appl..

[40]  Ashutosh Tiwari,et al.  The security challenges in the IoT enabled cyber-physical systems and opportunities for evolutionary computing & other computational intelligence , 2016, 2016 IEEE Congress on Evolutionary Computation (CEC).

[41]  Lovekesh Vig,et al.  LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection , 2016, ArXiv.

[42]  Fuad E. Alsaadi,et al.  Recent advances on filtering and control for cyber-physical systems under security and resource constraints , 2016, J. Frankl. Inst..

[43]  Yu Cheng,et al.  Deep Structured Energy Based Models for Anomaly Detection , 2016, ICML.

[44]  Mehmet K. Aktas,et al.  Emerging Security Mechanisms for Medical Cyber Physical Systems , 2016, IEEE/ACM Transactions on Computational Biology and Bioinformatics.

[45]  Nils Ole Tippenhauer,et al.  SWaT: a water treatment testbed for research and training on ICS security , 2016, 2016 International Workshop on Cyber-physical Systems for Smart Water Networks (CySWater).

[46]  Stanislav Ponomarev,et al.  Industrial Control System Network Intrusion Detection by Telemetry Analysis , 2016, IEEE Transactions on Dependable and Secure Computing.

[47]  Amr M. Youssef,et al.  Security Tradeoffs in Cyber Physical Systems: A Case Study Survey on Implantable Medical Devices , 2016, IEEE Access.

[48]  David K. Y. Yau,et al.  Data Driven Physical Modelling For Intrusion Detection In Cyber Physical Systems , 2016, SG-CRC.

[49]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[50]  Naixue Xiong,et al.  Design and Analysis of Multimodel-Based Anomaly Intrusion Detection Systems in Industrial Process Automation , 2015, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[51]  Hsiao-Hwa Chen,et al.  Intrusion Detection in Cyber-Physical Systems: Techniques and Challenges , 2014, IEEE Systems Journal.

[52]  Colleen Swanson,et al.  SoK: Security and Privacy in Implantable Medical Devices and Body Area Networks , 2014, 2014 IEEE Symposium on Security and Privacy.

[53]  Andreas Pitsillides,et al.  Survey in Smart Grid and Smart Home Security: Issues, Challenges and Countermeasures , 2014, IEEE Communications Surveys & Tutorials.

[54]  Ing-Ray Chen,et al.  A survey of intrusion detection techniques for cyber-physical systems , 2014, ACM Comput. Surv..

[55]  Stamatis Karnouskos,et al.  Stuxnet worm impact on industrial cyber-physical system security , 2011, IECON 2011 - 37th Annual Conference of the IEEE Industrial Electronics Society.

[56]  Edmund M. Clarke,et al.  Statistical Model Checking for Cyber-Physical Systems , 2011, ATVA.

[57]  Tak-Chung Fu,et al.  A review on time series data mining , 2011, Eng. Appl. Artif. Intell..

[58]  Igor Nai Fovino,et al.  A Multidimensional Critical State Analysis for Detecting Intrusions in SCADA Systems , 2011, IEEE Transactions on Industrial Informatics.

[59]  Mingyan Teng,et al.  Anomaly detection on time series , 2010, 2010 IEEE International Conference on Progress in Informatics and Computing.

[60]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[61]  Nathalie Japkowicz,et al.  A Mixture-of-Experts Framework for Learning from Imbalanced Data Sets , 2001, IDA.