Vulnerabilities and safety assurance methods in Cyber-Physical Systems: A comprehensive review

Abstract As Cyber-Physical Systems (CPSs) are a class of systems advancing in a number of safety-critical application areas, it is crucial to ensure that they operate without causing any harm to people, environment and assets. The complexity of CPSs though, render them vulnerable and accident-prone. In this study, the sources of complexity are meticulously examined and the state-of-the-art and novel methods that are used for the safety assurance of CPSs are reviewed. Furthermore, the identified safety assurance methods are assessed for their compatibility with the technical processes during the system design phase and the methods effectiveness on addressing the different CPSs sources of complexity is investigated. Advantages and disadvantages of the different safety assurance methods are also presented. Based on the results of this review, directions for the safety enhancement of CPSs and topics for future research in the area of CPSs safety are provided.

[1]  John A. McDermid,et al.  Hierarchically Performed Hazard Origin and Propagation Studies , 1999, SAFECOMP.

[2]  Dimitri Bohlender,et al.  Design and Verification of Restart-Robust Industrial Control Software , 2018, IFM.

[3]  Sakir Sezer,et al.  STPA-SafeSec: Safety and security analysis for cyber-physical systems , 2017, J. Inf. Secur. Appl..

[4]  Diego Mandelli,et al.  Probabilistic risk assessment modeling of digital instrumentation and control systems using two dynamic methodologies , 2010, Reliab. Eng. Syst. Saf..

[5]  Mohamed Sallak,et al.  A quantitative model for the risk evaluation of driver-ADAS systems under uncertainty , 2017, Reliab. Eng. Syst. Saf..

[6]  Jérémie Guiochet,et al.  Hazard analysis of human-robot interactions with HAZOP-UML , 2016, Safety Science.

[7]  Ronald L. Boring,et al.  Fifty Years of THERP and Human Reliability Analysis , 2012 .

[8]  Jérémie Guiochet,et al.  Can Robot Navigation Bugs Be Found in Simulation? An Exploratory Study , 2017, 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[9]  Enrico Zio,et al.  A Monte Carlo-based exploration framework for identifying components vulnerable to cyber threats in nuclear power plants , 2018, Reliab. Eng. Syst. Saf..

[10]  Fabio De Felice,et al.  A Hybrid Probabilistic Model for Evaluating and Simulating Human Error in Industrial Emergency Conditions (HEIE) , 2017, Journal of Failure Analysis and Prevention.

[11]  Robert C. Armstrong,et al.  Theorem-Proving Analysis of Digital Control Logic Interacting with Continuous Dynamics , 2015, NSV.

[12]  Satyandra K. Gupta,et al.  Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehicles , 2018, J. Syst. Softw..

[13]  Aditya P. Mathur,et al.  Aligning Cyber-Physical System Safety and Security , 2014, CSDM Asia.

[14]  Sridhar Adepu,et al.  Integrating Six-Step Model with Information Flow Diagrams for Comprehensive Analysis of Cyber-Physical System Safety and Security , 2017, 2017 IEEE 18th International Symposium on High Assurance Systems Engineering (HASE).

[15]  Min Wu,et al.  Safety Verification of Deep Neural Networks , 2016, CAV.

[16]  Julien Delange,et al.  Validating Safety and Security Requirements for Partitioned Architectures , 2009, Ada-Europe.

[17]  Peter H. Feiler,et al.  Dependability Modeling with the Architecture Analysis & Design Language (AADL) , 2007 .

[18]  Brian F. Gore,et al.  Man–machine Integration Design and Analysis System (MIDAS) v5: Augmentations, Motivations, and Directions for Aeronautics Applications , 2011 .

[19]  Christine Julien,et al.  Perceptions on the State of the Art in Verification and Validation in Cyber-Physical Systems , 2017, IEEE Systems Journal.

[20]  Enrico Zio,et al.  Challenges in the vulnerability and risk analysis of critical infrastructures , 2016, Reliab. Eng. Syst. Saf..

[21]  Mark Austin,et al.  Ontologies of Time and Time-based Reasoning for MBSE of Cyber-Physical Systems , 2013, CSER.

[22]  Ludovic Piètre-Cambacédès,et al.  A survey of approaches combining safety and security for industrial control systems , 2015, Reliab. Eng. Syst. Saf..

[23]  Selva S. Rivera,et al.  A novel qualitative prospective methodology to assess human error during accident sequences , 2018 .

[24]  Birgit Vogel-Heuser,et al.  Design, modelling, simulation and integration of cyber physical systems: Methods and applications , 2016, Comput. Ind..

[25]  Yu Peng,et al.  Review on cyber-physical systems , 2017, IEEE/CAA Journal of Automatica Sinica.

[26]  Steven Drager,et al.  Cyber-Physical Specification Mismatches , 2018, ACM Trans. Cyber Phys. Syst..

[27]  Martin Höst,et al.  Comparison of the FMEA and STPA safety analysis methods–a case study , 2019, Software Quality Journal.

[28]  Noureddine Zerhouni,et al.  Continuous validation of the PHM function in aircraft industry , 2015, 2015 First International Conference on Reliability Systems Engineering (ICRSE).

[29]  Wolfgang Reif,et al.  Runtime Model-Based Safety Analysis of Self-Organizing Systems with S# , 2015, 2015 IEEE International Conference on Self-Adaptive and Self-Organizing Systems Workshops.

[30]  Ayan Banerjee,et al.  Validation, Verification, and Formal Methods for Cyber-Physical Systems , 2017 .

[31]  D. L. Simms,et al.  Normal Accidents: Living with High-Risk Technologies , 1986 .

[32]  Jonghyun Kim,et al.  Systematic development of scenarios caused by cyber-attack-induced human errors in nuclear power plants , 2017, Reliab. Eng. Syst. Saf..

[33]  Markus Borg,et al.  An Industrial Survey of Safety Evidence Change Impact Analysis Practice , 2016, IEEE Transactions on Software Engineering.

[34]  Nikolaos Papakonstantinou,et al.  Common cause failure analysis of cyber–physical systems situated in constructed environments , 2013, Research in Engineering Design.

[35]  Luiz S. Martins-Filho,et al.  Processor-in-the-Loop Simulations Applied to the Design and Evaluation of a Satellite Attitude Control , 2014 .

[36]  Nancy G. Leveson,et al.  Complexity and Safety , 2011, CSDM.

[37]  Stefan Kowalewski,et al.  Model-in-the-Loop and Software-in-the-Loop Testing of Closed-Loop Automotive Software with Arttest , 2017, GI-Jahrestagung.

[38]  Ronald L. Boring.  How Many Performance Shaping Factors are Necessary for Human Reliability Analysis , 2010 .

[39]  Suman Jana,et al.  DeepTest: Automated Testing of Deep-Neural-Network-Driven Autonomous Cars , 2017, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[40]  John A. McDermid,et al.  An integrated tool set for software safety analysis , 1993, J. Syst. Softw..

[41]  Joanne Bechta Dugan,et al.  ANALYZING DYNAMIC FAULT TREES DERIVED FROM MODEL-BASED SYSTEM ARCHITECTURES , 2008 .

[42]  Wei Zheng,et al.  Formal model-based quantitative safety analysis using timed Coloured Petri Nets , 2018, Reliab. Eng. Syst. Saf..

[43]  Ievgen Ivanov,et al.  On Formalization of Semantics of Real-Time and Cyber-Physical Systems , 2018 .

[44]  Bradley R. Schmerl,et al.  Supporting Heterogeneity in Cyber-Physical Systems Architectures , 2014, IEEE Transactions on Automatic Control.

[45]  Tony Gorschek,et al.  Requirements engineering for safety-critical systems: A systematic literature review , 2016, Inf. Softw. Technol..

[46]  Sanja Dogramadzi,et al.  Environmental Hazard Analysis - a Variant of Preliminary Hazard Analysis for Autonomous Mobile Robots , 2014, J. Intell. Robotic Syst..

[47]  Dietmar P. F. Möller,et al.  Guide to Computing Fundamentals in Cyber-Physical Systems , 2016, Computer Communications and Networks.

[48]  Ning Ge,et al.  Integrated formal verification of safety-critical software , 2017, International Journal on Software Tools for Technology Transfer.

[49]  Shiwei Wang,et al.  A cognitive reliability model research for complex digital human-computer interface of industrial system , 2017, Safety Science.

[50]  Paul Pettersson,et al.  Analyzing a wind turbine system: From simulation to formal verification , 2017, Sci. Comput. Program..

[51]  Man Zhang,et al.  Uncertainty-Wise Cyber-Physical System test modeling , 2019, Software & Systems Modeling.

[52]  Joost-Pieter Katoen,et al.  Spacecraft early design validation using formal methods , 2014, Reliab. Eng. Syst. Saf..

[53]  Jean-Marie Flaus,et al.  A safety/security risk analysis approach of Industrial Control Systems: A cyber bowtie - combining new version of attack tree with bowtie analysis , 2018, Comput. Secur..

[54]  Hans J. Pasman,et al.  Process hazard analysis, hazard identification and scenario definition: Are the conventional tools sufficient, or should and can we do much better? , 2017 .

[55]  André Platzer.  Logic & Proofs for Cyber-Physical Systems , 2016, IJCAR.

[56]  Stefan Wagner,et al.  A Software Safety Verification Method Based on System-Theoretic Process Analysis , 2014, SAFECOMP Workshops.

[57]  Jose Ignacio Aizpurua.  Design of Dependable Systems: An Overview of Analysis and Verification Approaches , 2012 .

[58]  Ali Mosleh,et al.  Cognitive modeling and dynamic probabilistic simulation of operating crew response to complex system accidents: Part 1: Overview of the IDAC Model , 2007, Reliab. Eng. Syst. Saf..

[59]  Thomas Kuhn,et al.  Integration of Component Fault Trees into the UML , 2010, MoDELS.

[60]  Jérémie Guiochet,et al.  Safety-critical advanced robots: A survey , 2017, Robotics Auton. Syst..

[61]  Marvin Rausand,et al.  Defining complexity for risk assessment of sociotechnical systems: A conceptual framework , 2014 .

[62]  Shaojun Li,et al.  Study on Generation of Fault Trees from Altarica Models , 2014 .

[63]  Mark R. Blackburn,et al.  Modeling and cross-domain dependability analysis of cyber-physical systems , 2018, 2018 Annual IEEE International Systems Conference (SysCon).

[64]  Augusto Sampaio,et al.  Sound conformance testing for cyber-physical systems: Theory and implementation , 2017, Sci. Comput. Program..

[65]  Mohammad Reza Mousavi,et al.  Model-Based Testing of Cyber-Physical Systems , 2017 .

[66]  Paula Gonçalves,et al.  Unmanned aerial vehicle safety assessment modelling through petri Nets , 2017, Reliab. Eng. Syst. Saf..

[67]  Jean-Yves Choley,et al.  Topology-based Safety Analysis for Safety Critical CPS , 2016 .

[68]  Christoph Schmittner,et al.  A Case Study of FMVEA and CHASSIS as Safety and Security Co-Analysis Method for Automotive Cyber-physical Systems , 2015, CPSS@ASIACSS.

[69]  Qi Gong,et al.  Model-based System Safety Assessment of Aircraft Power Plant☆ , 2014 .

[70]  Philip Koopman,et al.  Monitor Based Oracles for Cyber-Physical System Testing: Practical Experience Report , 2014, 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks.

[71]  John P. Thomas,et al.  Extending and automating a systems-theoretic hazard analysis for requirements generation and analysis , 2013 .

[72]  Septavera Sharvia,et al.  A synthesis of logic and bio-inspired techniques in the design of dependable systems , 2016, Annu. Rev. Control..

[73]  Nicola Paoletti,et al.  Closed-Loop Quantitative Verification of Rate-Adaptive Pacemakers , 2018, ACM Trans. Cyber Phys. Syst..

[74]  Christel Baier,et al.  Principles of model checking , 2008 .

[75]  A. Platzer,et al.  ModelPlex: verified runtime validation of verified cyber-physical system models , 2016, Formal Methods Syst. Des..

[76]  Joost-Pieter Katoen,et al.  The COMPASS Approach: Correctness, Modelling and Performability of Aerospace Systems , 2009, SAFECOMP.

[77]  Knut Åkesson,et al.  Using Valued Booleans to Find Simpler Counterexamples in Random Testing of Cyber-Physical Systems , 2018 .

[78]  Jan Erik Vinnem,et al.  Deriving verification objectives and scenarios for maritime systems using the systems-theoretic process analysis , 2018, Reliab. Eng. Syst. Saf..

[79]  Wei Li,et al.  Modelling and Verification for Swarm Robotics , 2018, IFM.

[80]  Floris Goerlandt,et al.  Validity and validation of safety-related quantitative risk analysis , 2016 .

[81]  Antoine Rauzy,et al.  The AltaRica Formalism for Describing Concurrent Systems , 1999, Fundam. Informaticae.

[82]  Brian J. Phillips,et al.  Verification Points for Self-adaptive Systems , 2014, Complex Adaptive Systems.

[83]  Alexander Knapp,et al.  Qualitative and quantitative analysis of safety-critical systems with , 2017, International Journal on Software Tools for Technology Transfer.

[84]  Bert Bos,et al.  Co-modelling of Faults and Fault Tolerance Mechanisms , 2014, Collaborative Design for Embedded Systems.

[85]  Jan Erik Vinnem,et al.  A systems approach to risk analysis of maritime operations , 2017 .

[86]  Zhuzhong Qian,et al.  Modeling and checking for Cyber-Physical System based on hybrid interface automata , 2015, Pervasive Mob. Comput..

[87]  Marco Bozzano,et al.  The xSAP Safety Analysis Platform , 2016, TACAS.

[88]  Jinkyun Park,et al.  Use of a big data analysis technique for extracting HRA data from event investigation reports based on the Safety-II concept , 2020, Reliab. Eng. Syst. Saf..

[89]  Zahid H. Qureshi,et al.  A review of accident modelling approaches for complex socio-technical systems , 2007 .

[90]  Homayoon Dezfuli,et al.  NASA System Safety Handbook. Volume 1; System Safety Framework and Concepts for Implementation , 2011 .

[91]  Panganamala Ramana Kumar,et al.  Cyber–Physical Systems: A Perspective at the Centennial , 2012, Proceedings of the IEEE.

[92]  Frank Hearl,et al.  Working safely with robot workers: Recommendations for the new workplace , 2016, Journal of occupational and environmental hygiene.

[93]  Roberto Passerone,et al.  Dependability Assessment of SOA-Based CPS With Contracts and Model-Based Fault Injection , 2018, IEEE Transactions on Industrial Informatics.

[94]  Insup Lee,et al.  Challenges and Research Directions in Medical Cyber–Physical Systems , 2012, Proceedings of the IEEE.

[95]  Sebastian Engell,et al.  Core Research and Innovation Areas in Cyber-Physical Systems of Systems , 2015, ERCIM News.

[96]  Raffaele Iannone,et al.  A Simulator for Human Error Probability Analysis (SHERPA) , 2015, Reliab. Eng. Syst. Saf..

[97]  Fabio De Felice,et al.  Development of a risk analysis model to evaluate human error in industrial plants and in critical infrastructures , 2017 .

[98]  Frank Vahid,et al.  A Survey on Concepts, Applications, and Challenges in Cyber-Physical Systems , 2014, KSII Trans. Internet Inf. Syst..

[99]  David D. Walden,et al.  Systems engineering handbook : a guide for system life cycle processes and activities , 2015 .

[100]  Osman Hasan,et al.  Formal Verification of Cyber-Physical Systems: Coping with Continuous Elements , 2013, ICCSA.

[101]  Septavera Sharvia,et al.  Model-based dependability analysis , 2016 .

[102]  Andrew Lewis,et al.  Systems-Theoretic Safety Assessment of Robotic Telesurgical Systems , 2015, SAFECOMP.

[103]  Yanhua Zou,et al.  An assessment method of operator’s situation awareness reliability based on fuzzy logic-AHP , 2019, Safety Science.

[104]  Nancy G. Leveson,et al.  Engineering a Safer World: Systems Thinking Applied to Safety , 2012 .

[105]  Laurent Wouters,et al.  Ensuring Dependability and Performance for CPS Design , 2017 .

[106]  Mehrdad Sabetzadeh,et al.  An extended systematic literature review on provision of evidence for safety certification , 2014, Inf. Softw. Technol..

[107]  Peter Liggesmeyer,et al.  A New Component Concept for Fault Trees , 2003, SCS.

[108]  Udo Lindemann,et al.  Integrated Matrix-based Fault Tree Generation and Evaluation☆ , 2015 .

[109]  Ellen J. Bass,et al.  Using Formal Verification to Evaluate Human-Automation Interaction: A Review , 2013, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[110]  Radha Poovendran,et al.  Aviation Cyber–Physical Systems: Foundations for Future Aircraft and Air Transport , 2013, Proceedings of the IEEE.

[111]  Swu Yih,et al.  A simulation platform for human-machine interaction safety analysis of cyber-physical systems , 2018, International Journal of Industrial Ergonomics.

[112]  Marilyn Wolf,et al.  Safety and Security in Cyber-Physical Systems and Internet-of-Things Systems , 2018, Proceedings of the IEEE.

[113]  Masooda Bashir,et al.  Trust in Automation , 2015, Hum. Factors.

[114]  Giedre Sabaliauskaite,et al.  Integrating Autonomous Vehicle Safety and Security Analysis Using STPA Method and the Six-Step Model , 2018 .

[115]  Fiona Robertson,et al.  The Johari Window , 2016 .

[116]  John Thomas,et al.  Integration of Multiple Active Safety Systems using STPA , 2015 .

[117]  Edmund M. Clarke,et al.  Statistical Model Checking for Cyber-Physical Systems , 2011, ATVA.

[118]  Davide Bresolin,et al.  Formal verification of robotic surgery tasks by reachability analysis , 2015, Microprocess. Microsystems.

[119]  Septavera Sharvia,et al.  Integrating model checking with HiP-HOPS in model-based safety analysis , 2015, Reliab. Eng. Syst. Saf..

[120]  David B. Kaber,et al.  Enhanced Hazard Analysis and Risk Assessment for Human-in-the-Loop Systems , 2017, Hum. Factors.

[121]  Tiedo Tinga,et al.  Improving failure analysis efficiency by combining FTA and FMEA in a recursive manner , 2018, Reliab. Eng. Syst. Saf..

[122]  Lionel C. Briand,et al.  Test Generation and Test Prioritization for Simulink Models with Dynamic Behavior , 2019, IEEE Transactions on Software Engineering.