Can Robot Navigation Bugs Be Found in Simulation? An Exploratory Study

The ability to navigate in diverse and previously unknown environments is a critical service of autonomous robots. The validation of the navigation software typically involves test campaigns in the field, which are costly and potentially risky for the robot itself or its environment. An alternative approach is to perform simulation-based testing, by immersing the software in virtual worlds. A question is then whether the bugs revealed in real worlds can also be found in simulation. The paper reports on an exploratory study of bugs in an academic software for outdoor robots navigation. The detailed analysis of the triggers and effects of these bugs shows that most of them can be revealed in low-fidelity simulation. It also provides insights into interesting navigation scenarios to test as well as into how to address the test oracle problem.

[1]  Matthieu Herrb,et al.  GenoM3: Building middleware-independent robotic components , 2010, 2010 IEEE International Conference on Robotics and Automation.

[2]  Roland Siegwart,et al.  Designing a secure and robust mobile interacting robot for the long term , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[3]  Julian Togelius,et al.  Search-Based Procedural Content Generation: A Taxonomy and Survey , 2011, IEEE Transactions on Computational Intelligence and AI in Games.

[4]  J.J. Biesiadecki,et al.  The Mars Exploration Rover surface mobility flight software driving ambition , 2006, 2006 IEEE Aerospace Conference.

[5]  Andrew Howard,et al.  Design and use paradigms for Gazebo, an open-source multi-robot simulator , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[6]  Andrew Rae,et al.  Situation coverage – a coverage criterion for testing autonomous robots , 2015 .

[7]  Iago Abal,et al.  42 variability bugs in the linux kernel: a qualitative analysis , 2014, ASE.

[8]  Anneliese Amschler Andrews,et al.  World Model for Testing Autonomous Systems Using Petri Nets , 2016, 2016 IEEE 17th International Symposium on High Assurance Systems Engineering (HASE).

[9]  Gerald Steinbauer,et al.  Improving dependability of industrial transport robots using model-based techniques , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[10]  Ubbo Visser,et al.  RoboCup: 10 Years of Achievements and Future Challenges , 2007, AI Mag..

[11]  Kishor S. Trivedi,et al.  Reproducibility of Environment-Dependent Software Failures: An Experience Report , 2014, 2014 IEEE 25th International Symposium on Software Reliability Engineering.

[12]  John McDermid,et al.  Safety Validation of Sense and Avoid Algorithms Using Simulation and Evolutionary Search , 2014, SAFECOMP.

[13]  Gerald Steinbauer,et al.  A Survey about Faults of Robots Used in RoboCup , 2012, RoboCup.

[14]  Jérémie Guiochet,et al.  Virtual Worlds for Testing Robot Navigation: A Study on the Difficulty Level , 2016, 2016 12th European Dependable Computing Conference (EDCC).

[15]  Thierry Siméon,et al.  Motion generation for a rover on rough terrains , 2001, Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the the Next Millennium (Cat. No.01CH37180).

[16]  Zoltán Szatmári,et al.  A Concept for Testing Robustness and Safety of the Context-Aware Behaviour of Autonomous Systems , 2012, KES-AMSTA.

[17]  Andrew Vardy,et al.  A survey of AUV and robot simulators for multi-vehicle operations , 2014, 2014 IEEE/OES Autonomous Underwater Vehicles (AUV).

[18]  Michael D. Bond,et al.  Tracking bad apples: reporting the origin of null and undefined value errors , 2007, OOPSLA.

[19]  James Arnold,et al.  Testing Autonomous Robot Control Software Using Procedural Content Generation , 2013, SAFECOMP.

[20]  Gilberto Echeverria,et al.  Modular open robots simulation engine: MORSE , 2011, 2011 IEEE International Conference on Robotics and Automation.

[21]  Inderpal S. Bhandari,et al.  Orthogonal Defect Classification - A Concept for In-Process Measurements , 1992, IEEE Trans. Software Eng..

[22]  Victor R. Basili,et al.  Identifying domain-specific defect classes using inspections and change history , 2006, ISESE '06.

[23]  John McDermid,et al.  On the Validation of a UAV Collision Avoidance System Developed by Model-Based Optimization: Challenges and a Tentative Partial Solution , 2016, 2016 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshop (DSN-W).

[24]  Shan Lu,et al.  Understanding and detecting real-world performance bugs , 2012, PLDI.