Design Considerations for Building Credible Security Testbeds: A Systematic Study of Industrial Control System Use Cases

This paper presents a mapping framework for design factors and implementation process for building credible Industrial Control Systems (ICS) security testbeds. The resilience of ICSs has become a critical concern to operators and governments following widely publicised cyber security events. The inability to apply conventional Information Technology security practice to ICSs further compounds challenges in adequately securing critical systems. To overcome these challenges, and do so without impacting live environments, testbeds for the exploration, development and evaluation of security controls are widely used. However, how a testbed is designed and its attributes, can directly impact not only its viability but also its credibility as a whole. Through a combined systematic and thematic analysis and mapping of ICS security testbed design attributes, this paper suggests that the expertise of human experimenters, design objectives, the implementation approach, architectural coverage, core characteristics, and evaluation methods; are considerations that can help establish or enhance confidence, trustworthiness and acceptance; thus, credibility of ICS security testbeds.

[1]  Ting Wang,et al.  An Industrial Control System Testbed Based on Emulation, Physical Devices and Simulation , 2014, Critical Infrastructure Protection.

[2]  S. Y. Harmon,et al.  Evolving the validation process maturity model (VPMM) , 2008 .

[3]  Erik Westring,et al.  A Survey of Industrial Control System Testbeds , 2015, NordSec.

[4]  Joseph H. Morrison,et al.  Towards a Credibility Assessment of Models and Simulations , 2008 .

[5]  David M. Nicol,et al.  The Virtual Power System Testbed and Inter-Testbed Integration , 2009, CSET.

[6]  Karen A. Scarfone,et al.  Guide to Industrial Control Systems (ICS) Security , 2015 .

[7]  Bryan Richardson,et al.  Supervisory Command and Data Acquisition (SCADA) system cyber security analysis using a live, virtual, and constructive (LVC) testbed , 2012, MILCOM 2012 - 2012 IEEE Military Communications Conference.

[8]  Awais Rashid,et al.  A Reference Architecture for IIoT and Industrial Control Systems Testbeds , 2019, Living in the Internet of Things (IoT 2019).

[9]  Igor Nai Fovino,et al.  An experimental platform for assessing SCADA vulnerabilities and countermeasures in power plants , 2010, 3rd International Conference on Human System Interaction.

[10]  Maria J Grant,et al.  A typology of reviews: an analysis of 14 review types and associated methodologies. , 2009, Health information and libraries journal.

[11]  Chris Hankin,et al.  Open Testbeds for CNI , 2018 .

[12]  Yuval Elovici,et al.  Security Testbed for the Internet of Things , 2016, ArXiv.

[13]  David Hutchison,et al.  A survey of cyber security management in industrial control systems , 2015, Int. J. Crit. Infrastructure Prot..

[15]  André Thomas,et al.  Contribution to reusability and modularity of manufacturing systems simulation models: Application to distributed control simulation within DFT context , 2008 .

[16]  Jon Davis,et al.  A Survey of Cyber Ranges and Testbeds , 2013 .

[17]  Béla Genge,et al.  Cyber-physical testbeds , 2014, CACM.

[18]  Wei Zhao,et al.  Testbed techniques of industrial control system , 2013, Proceedings of 2013 3rd International Conference on Computer Science and Network Technology.

[19]  David A. Cook.  How to Perform Credible Verification , Validation , and Accreditation for Modeling and Simulation , 2005 .

[20]  Richard Candell,et al.  An Industrial Control System Cybersecurity Performance Testbed , 2015 .

[21]  Charles R. McLean,et al.  Modeling and Simulation of Critical Infrastructure Systems for Homeland Security Applications , 2011 .

[22]  David Hutchison,et al.  Achieving ICS Resilience and Security through Granular Data Flow Management , 2016, CPS-SPC '16.

[23]  Thomas D. Hedberg,et al.  Design and configuration of the smart manufacturing systems test bed , 2017 .

[24]  David Hutchison,et al.  Pains, Gains and PLCs: Ten Lessons from Building an Industrial Control Systems Testbed for Security Research , 2017, CSET @ USENIX Security Symposium.

[25]  Timothy G. Trucano,et al.  Predictive Capability Maturity Model for computational modeling and simulation. , 2007 .

[26]  Anders Skoogh,et al.  Cyber-Physical Production Testbed: Literature Review and Concept Development , 2018 .

[27]  Timea Pahi,et al.  Design Considerations for Cyber Security Testbeds: A Case Study on a Cyber Security Testbed for Education , 2017, 2017 IEEE 15th Intl Conf on Dependable, Autonomic and Secure Computing, 15th Intl Conf on Pervasive Intelligence and Computing, 3rd Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress(DASC/PiCom/DataCom/CyberSciTech).

[28]  Bryan T. Richardson,et al.  Position Paper : Modeling and Simulation for Process Control System Cyber Security Research , Development and Applications , 2009 .

[29]  Michail Maniatakos,et al.  The Cybersecurity Landscape in Industrial Control Systems , 2016, Proceedings of the IEEE.

[30]  Jack P. C. Kleijnen,et al.  EUROPEAN JOURNAL OF OPERATIONAL , 1992 .

[31]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[32]  Eric A. M. Luiijf,et al.  Creating a European SCADA Security Testbed , 2007, Critical Infrastructure Protection.

[33]  Zahir Tari,et al.  SCADAVT-A framework for SCADA security testbed based on virtualization technology , 2013, 38th Annual IEEE Conference on Local Computer Networks.

[34]  Aditya Ashok,et al.  Cyber-Physical Security Testbeds: Architecture, Application, and Evaluation for Smart Grid , 2013, IEEE Transactions on Smart Grid.

[35]  Awais Rashid,et al.  Oops I Did it Again: Further Adventures in the Land of ICS Security Testbeds , 2019, CPS-SPC@CCS.

[36]  Jose J. Padilla,et al.  A characterization of cybersecurity simulation scenarios , 2016, SpringSim.

[37]  Theodore J. Williams,et al.  The Purdue Enterprise Reference Architecture , 1992, DIISM.

[38]  Vincent Urias,et al.  Performing cyber security analysis using a live, virtual, and constructive (LVC) testbed , 2010, 2010 - MILCOM 2010 MILITARY COMMUNICATIONS CONFERENCE.

[39]  Vincent Urias,et al.  Cyber security analysis testbed: Combining real, emulation, and simulation , 2010, 44th Annual 2010 IEEE International Carnahan Conference on Security Technology.

[40]  Alexander Gluhak,et al.  A survey on facilities for experimental internet of things research , 2011, IEEE Communications Magazine.

[41]  Averill M. Law,et al.  How to build valid and credible simulation models , 2008, 2008 Winter Simulation Conference.