Integration testing for robotic systems

Conventional approaches for the validation of robotic systems include simulations and functional software testing. Although these approaches are interesting, they do not offer coverage information to guide the testing activity. In this sense, the introduction of coverage testing to the validation of mobile robotic systems seems to promise. This paper proposes a integration testing approach for robotic systems in a simulation environment. The approach is composed of test scenarios, constructed based on functional testing, and structural testing towards revealing defects and providing a better systematization of the testing activity in this application domain. The proposed approach focuses on systems developed in ROS, the robot operating system, in which communication can be established through a publish/subscribe interaction schema. The paper provides an example of the way the integration testing approach can be applied to robotic systems. An experimental study conducted with developers of mobile robotic systems evaluated the gains and proved the applicability of the approach to the industry. The results confirmed its advantages for the integration testing of mobile robotic systems. It can check functional behavior (test scenarios), or reach a high structural coverage (coverage criteria), and reveal common defects in mobile robotic systems.

[1]  Cem Kaner,et al.  An Introduction to Scenario Testing , 2013 .

[2]  Ana R. Cavalli,et al.  InRob: An approach for testing interoperability and robustness of real-time embedded software , 2012, J. Syst. Softw..

[3]  Hara Gopal Mani Pakala,et al.  Integration Testing of Multiple Embedded Processing Components , 2011 .

[4]  Spyros G. Tzafestas,et al.  Introduction to Mobile Robot Control , 2013 .

[5]  Denis Fernando Wolf,et al.  Structural Testing of Autonomous Vehicles , 2013, International Conference on Software Engineering and Knowledge Engineering.

[6]  Hardi Hungar,et al.  Using contract-based component specifications for virtual integration testing and architecture design , 2011, 2011 Design, Automation & Test in Europe.

[7]  Hong Seong Park,et al.  Web-based automated black-box testing framework for component based robot software , 2012, UbiComp '12.

[8]  A. Jefferson Offutt,et al.  Introduction to Software Testing , 2008 .

[9]  Jérémie Guiochet,et al.  Can Robot Navigation Bugs Be Found in Simulation? An Exploratory Study , 2017, 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS).

[10]  Gerald Steinbauer,et al.  A Survey about Faults of Robots Used in RoboCup , 2012, RoboCup.

[11]  Anne-Marie Kermarrec,et al.  The many faces of publish/subscribe , 2003, CSUR.

[12]  Bo Sun,et al.  Research of embedded software testing method based on AADL modes , 2011, 2011 IEEE 3rd International Conference on Communication Software and Networks.

[13]  Paulo Sergio Lopes de Souza,et al.  ValiPar Service: Structural Testing of Concurrent Programs as a Web Service Composition , 2016 .

[14]  Hong-Seong Park,et al.  A Hierarchical Test Model and Automated Test Framework for RTC , 2009, FGIT.

[15]  Silvia Regina Vergilio,et al.  Structural testing criteria for message-passing parallel programs , 2008 .

[16]  Ana Cavalli,et al.  A Pragmatic Approach to Generating Test Sequences for Embedded Systems , 1997 .

[17]  Elaine J. Weyuker,et al.  Selecting Software Test Data Using Data Flow Information , 1985, IEEE Transactions on Software Engineering.

[18]  Nicolas Anquetil,et al.  Test case selection in industry: an analysis of issues related to static approaches , 2016, Software Quality Journal.

[19]  Ed Zaluska,et al.  Structural testing for message‐passing concurrent programs: an extended test model , 2014, Concurr. Comput. Pract. Exp..

[20]  Robin R. Murphy,et al.  Follow-up analysis of mobile robot failures , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[21]  Cecília M. F. Rubira,et al.  Architecting Fault Tolerance with Exception Handling: Verification and Validation , 2009, Journal of Computer Science and Technology.

[22]  Mehmet Söylemez,et al.  Challenges of software process and product quality improvement: catalyzing defect root-cause investigation by process enactment data analysis , 2016, Software Quality Journal.

[23]  Luciano Baresi,et al.  An Introduction to Software Testing , 2006, FoVMT.

[24]  Roland Siegwart,et al.  Introduction to Autonomous Mobile Robots , 2004 .

[25]  Jang-Eui Hong,et al.  Investigating relationships between functional coupling and the energy efficiency of embedded software , 2016, Software Quality Journal.

[26]  Simone do Rocio Senger de Souza,et al.  Integration testing criteria for mobile robotic systems , 2015, SEKE.

[27]  Simone do Rocio Senger de Souza,et al.  Contributions for the structural testing of multithreaded programs: coverage criteria, testing tool, and experimental evaluation , 2017, Software Quality Journal.

[28]  Tae-Yong Kuc,et al.  Simulation Based Functional and Performance Evaluation of Robot Components and Modules , 2011, 2011 International Conference on Information Science and Applications.

[29]  Fevzi Belli,et al.  Communication Sequence Graphs for Mutation-Oriented Integration Testing , 2009, 2009 Third IEEE International Conference on Secure Software Integration and Reliability Improvement.

[30]  Hong Seong Park,et al.  RPIST: Required and provided interface specification-based test case generation and execution methodology for robot software component , 2011, 2011 8th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI).

[31]  Hareton Leung,et al.  A study of integration testing and software regression at the integration level , 1990, Proceedings. Conference on Software Maintenance 1990.

[32]  David S. Rosenblum,et al.  Detecting problematic message sequences and frequencies in distributed systems , 2012, OOPSLA '12.

[33]  Glenford J. Myers,et al.  Art of Software Testing , 1979 .

[34]  Jing Xiao,et al.  Model-based testing of real-time adaptive motion planning (RAMP) , 2016, 2016 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR).

[35]  Hong-Seong Park,et al.  SITAT: Simulation-based Interface Testing Automation Tool for Robot Software Component , 2010 .

[36]  Alexander Verl,et al.  Enhancing Software Portability with a Testing and Evaluation Platform , 2014, ISR 2014.

[37]  Hara Gopal Mani Pakala,et al.  Sensors integration in embedded systems , 2010, 2010 International Conference on Power, Control and Embedded Systems.

[38]  Zijiang Yang,et al.  HAVE: Detecting Atomicity Violations via Integrated Dynamic and Static Analysis , 2009, FASE.

[39]  Raja Chatila,et al.  On Fault Tolerance and Robustness in Autonomous Systems , 2004 .

[40]  David Garlan,et al.  Model Checking Publish-Subscribe Systems , 2003, SPIN.

[41]  Matthias Scheutz,et al.  Development environments for autonomous mobile robots: A survey , 2007, Auton. Robots.

[42]  Sung Deok Cha,et al.  Construction of global finite state machine for testing task interactions written in message sequence charts , 2002, SEKE '02.

[43]  Denis F. Wolf,et al.  Path Planning and Autonomous Navigation using AMCL and AD* , 2015, 2015 12th Latin American Robotics Symposium and 2015 3rd Brazilian Symposium on Robotics (LARS-SBR).