A Characterization of the Utility of Using Artificial Intelligence to Test Two Artificial Intelligence Systems

An artificial intelligence system, designed for operations in a real-world environment faces a nearly infinite set of possible performance scenarios. Designers and developers, thus, face the challenge of validating proper performance across both foreseen and unforeseen conditions, particularly when the artificial intelligence is controlling a robot that will be operating in close proximity, or may represent a danger, to humans. While the manual creation of test cases allows limited testing (perhaps ensuring that a set of foreseeable conditions trigger an appropriate response), this may be insufficient to fully characterize and validate safe system performance. An approach to validating the performance of an artificial intelligence system using a simple artificial intelligence test case producer (AITCP) is presented. The AITCP allows the creation and simulation of prospective operating scenarios at a rate far exceeding that possible by human testers. Four scenarios for testing an autonomous navigation control system are presented: single actor in two-dimensional space, multiple actors in two-dimensional space, single actor in three-dimensional space, and multiple actors in three-dimensional space. The utility of using the AITCP is compared to that of human testers in each of these scenarios.

[1]  Gilbert Laporte,et al.  A Tabu Search Heuristic for the Vehicle Routing Problem , 1991 .

[2]  Rodney A. Brooks,et al.  Artificial Life and Real Robots , 1992 .

[3]  Mark Harman,et al.  The role of Artificial Intelligence in Software Engineering , 2012, 2012 First International Workshop on Realizing AI Synergies in Software Engineering (RAISE).

[4]  Bharti Suri,et al.  Analyzing test case selection & prioritization using ACO , 2011, SOEN.

[5]  Fred Glover,et al.  Tabu Search: A Tutorial , 1990 .

[6]  Kenneth Morgan,et al.  Modified cuckoo search: A new gradient free optimisation algorithm , 2011 .

[7]  Matthew M. Huntbach,et al.  The Art in Artificial Intelligence , 1999 .

[8]  Rodney A. Brooks,et al.  Elephants don't play chess , 1990, Robotics Auton. Syst..

[9]  Radovan R. Bulatović,et al.  Cuckoo Search algorithm: A metaheuristic approach to solving the problem of optimum synthesis of a six-bar double dwell linkage , 2013 .

[10]  Ioan Salomie,et al.  A Hybrid Firefly-inspired Approach for Optimal Semantic Web Service Composition , 2011, Scalable Comput. Pract. Exp..

[11]  Paul C. Jorgensen,et al.  Software Testing: A Craftsman's Approach , 1995 .

[12]  Balakrishnan Chandrasekaran,et al.  On evaluating ai systems for medical diagnosis , 1989 .

[13]  Pattie Maes,et al.  Designing autonomous agents: Theory and practice from biology to engineering and back , 1990, Robotics Auton. Syst..

[14]  Amir Hossein Gandomi,et al.  Cuckoo search algorithm: a metaheuristic approach to solve structural optimization problems , 2011, Engineering with Computers.

[15]  Xin-She Yang,et al.  Cuckoo Search via Lévy flights , 2009, 2009 World Congress on Nature & Biologically Inspired Computing (NaBIC).

[16]  Jiang Wu,et al.  Novel intelligent water drops optimization approach to single UCAV smooth trajectory planning , 2009 .

[17]  Glenis Moore,et al.  The art of artificial intelligence , 1987 .

[18]  R. J. Beynon,et al.  Computers , 1985, Comput. Appl. Biosci..

[19]  Jonathan Schaeffer,et al.  The challenge of poker , 2002, Artif. Intell..

[20]  F. Glover.  HEURISTICS FOR INTEGER PROGRAMMING USING SURROGATE CONSTRAINTS , 1977 .

[21]  Rodney A. Brooks,et al.  Intelligence Without Reason , 1991, IJCAI.

[22]  R A Brooks,et al.  New Approaches to Robotics , 1991, Science.

[23]  Victor W. Marek,et al.  Computing with Default Logic , 1999, Artif. Intell..

[24]  Francesco Mondada,et al.  Evolution of neural control structures: some experiments on mobile robots , 1995, Robotics Auton. Syst..

[25]  Kerrie L. Mengersen,et al.  A proposed validation framework for expert elicited Bayesian Networks , 2013, Expert Syst. Appl..

[26]  Franz Wotawa,et al.  Debugging and test case generation using constraints and mutations , 2011, 2011 Proceedings of the Ninth International Workshop on Intelligent Solutions in Embedded Systems.

[27]  Adam Pettle.  Therac 25 , 2000 .

[28]  Hamed Shah-Hosseini,et al.  Problem solving by intelligent water drops , 2007, 2007 IEEE Congress on Evolutionary Computation.

[29]  Sami M. Halawani.  Safety Issues of Computer Failure , 2005, Egypt. Comput. Sci. J..

[30]  Peng Dai,et al.  Artificial Intelligence for Artificial Artificial Intelligence , 2011, AAAI.

[31]  M. O. Hawksford,et al.  Munich, Germany , 1997 .

[32]  Praveen Ranjan Srivastava,et al.  Test Case Optimization Using Artificial Bee Colony Algorithm , 2011, ACC.