Beyond Usability Evaluation: Analysis of Human-Robot Interaction at a Major Robotics Competition

Human-robot interaction (HRI) is a relatively new field of study. To date, most of the effort in robotics has been spent in developing hardware and software that expands the range of robot functionality and autonomy. In contrast, little effort has been spent so far to ensure that the robotic displays and interaction controls are intuitive for humans. This study applied robotics, human-computer interaction (HCI), and computer-supported cooperative work (CSCW) expertise to gain experience with HCI/CSCW evaluation techniques in the robotics domain. As a case study for this article, we analyzed four different robot systems that competed in the 2002 American Association for Artificial Intelligence Robot Rescue Competition. These systems completed urban search and rescue tasks in a controlled environment with predetermined scoring rules that provided objective measures of success. This study analyzed pre-evaluation questionnaires; videotapes of the robots, interfaces, and operators; maps of the robots' paths through the competition arena; post-evaluation debriefings; and critical incidents (e.g., when the robots damaged the test arena). As a result, this study developed guidelines for developing interfaces for HRI.

[1]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[2]  Holly A. Yanco.  Designing metrics for comparing the performance of robotic systems in robot competitions , 2002 .

[3]  Adam Jacoff,et al.  A Standard Test Course for Urban Search and Rescue Robots , 2000 .

[4]  Wolfram Burgard,et al.  Probabilistic Algorithms and the Interactive Museum Tour-Guide Robot Minerva , 2000, Int. J. Robotics Res..

[5]  Holly A. Yanco,et al.  A Taxonomy for Human-Robot Interaction , 2002 .

[6]  Nancy G Leveson,et al.  Software safety: why, what, and how , 1986, CSUR.

[7]  Jean Scholtz,et al.  Awareness in human-robot interactions , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[8]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[9]  Jakob Nielsen,et al.  Enhancing the explanatory power of usability heuristics , 1994, CHI '94.

[10]  Holly A. Yanco,et al.  Shared user-computer control of a robotic wheelchair system , 2000 .

[11]  Reid G. Simmons,et al.  GRACE: An Autonomous Robot for the AAAI Robot Challenge , 2003, AI Mag..

[12]  Charles E. Thorpe,et al.  Collaborative control: a robot-centric model for vehicle teleoperation , 2001 .

[13]  Robin R. Murphy,et al.  Human-robot interactions during the robot-assisted urban search and rescue response at the World Trade Center , 2003, IEEE Trans. Syst. Man Cybern. Part B.

[14]  Alex Meystel.  Measuring Performance and Intelligence of Intelligent Systems , 2000 .

[15]  David E. Kieras,et al.  Towards a Practical GOMS Model Methodology for User Interface Design , 1988 .

[16]  J. V. Draper,et al.  Next Generation Munitions Handler: Human-Machine Interface and Preliminary Performance Evaluation , 1999 .

[17]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[18]  Robin R. Murphy,et al.  Moonlight in Miami : A Field Study of Human-Robot Interaction in the Context of an Urban Search and Rescue Disaster Response Training Exercise , 2003 .

[19]  Jean C. Scholtz,et al.  Evaluation Methods for Human-System Performance of Intelligent Systems , 2002 .