Automated Unit Testing of a SCADA Control Software: An Industrial Case Study Based on Action Research

We report in this case-study paper our experience and success story with a practical approach and tool for unit regression testing of a SCADA (Supervisory Control and Data Acquisition) software. The tool uses a black-box specification of the units under test to automatically generate NUnit test code. We then improved the test suite by white-box and mutation testing. The approach and tool were developed in an action-research project to test a commercial large-scale SCADA system called Rocket.

[1]  Tore Dybå,et al.  Evidence-based software engineering , 2004, Proceedings. 26th International Conference on Software Engineering.

[2]  Vahid Garousi,et al.  An Open-Source Tool for Automated Generation of Black-Box xUnit Test Code and Its Industrial Evaluation , 2010, TAIC PART.

[3]  Stephen H. Edwards,et al.  Experiences evaluating the effectiveness of JML-JUnit testing , 2004, SOEN.

[4]  Vahid Garousi,et al.  Introducing Automated Environment Configuration Testing in an Industrial Setting , 2010, SEKE.

[5]  K. Hambridge.  Action research. , 2000, Professional nurse.

[6]  Nachiappan Nagappan,et al.  Evaluating the efficacy of test-driven development: industrial case studies , 2006, ISESE '06.

[7]  David E. Avison,et al.  Controlling action research projects , 2001, Inf. Technol. People.

[8]  Lionel C. Briand.  Useful software engineering research - leading a double-agent life , 2011, ICSM.

[9]  Mika Katara,et al.  Experiences of System-Level Model-Based GUI Testing of an Android Application , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[10]  Janice Singer,et al.  Guide to Advanced Empirical Software Engineering , 2007 .

[11]  Jing Guan,et al.  An industrial case study of structural testing applied to safety-critical embedded software , 2006, ISESE '06.

[12]  Shuang Wang,et al.  Comparison of Unit-Level Automated Test Generation Tools , 2009, 2009 International Conference on Software Testing, Verification, and Validation Workshops.

[13]  Ken Johnston,et al.  How We Test Software at Microsoft , 2008 .

[14]  Vahid Garousi,et al.  A replicated survey of software testing practices in the Canadian province of Alberta: What has changed from 2004 to 2009? , 2010, J. Syst. Softw..

[15]  Atul Gupta,et al.  An approach for experimentally evaluating effectiveness and efficiency of coverage criteria for software testing , 2008, International Journal on Software Tools for Technology Transfer.

[16]  Eduardo Aranha,et al.  Model Based Test Generation : An Industrial Experience , 2007 .

[17]  Lionel C. Briand,et al.  An enhanced test case selection approach for model-based testing: an industrial case study , 2010, FSE '10.

[18]  Stefan Biffl,et al.  Value-Based Management of Software Testing , 2006, Value-Based Software Engineering.

[19]  Jacek Czerwonka,et al.  CRANE: Failure Prediction, Change Analysis and Test Prioritization in Practice -- Experiences from Windows , 2011, 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation.

[20]  Lionel C. Briand,et al.  Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria , 2006, IEEE Transactions on Software Engineering.

[21]  Cem Kaner,et al.  Foundations of Software Testing , 2013 .

[22]  Thomas J. Ostrand,et al.  Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria , 1994, Proceedings of 16th International Conference on Software Engineering.

[23]  Tony Gorschek,et al.  A Model for Technology Transfer in Practice , 2006, IEEE Software.

[24]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[25]  Lionel C. Briand,et al.  Industrial experiences with automated regression testing of a legacy database application , 2011, 2011 27th IEEE International Conference on Software Maintenance (ICSM).

[26]  Guilherme Horta Travassos,et al.  Action research use in software engineering: An initial survey , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.