Human control of cooperating robots

Advances in robotic technologies and artificial intelligence are allowing robots to emerge from research laboratories into our lives. Experiences with field applications show that we have underestimated the importance of human-robot interaction (HRI) and that new problems arise in HRI as robotic technologies expand. This thesis classifies HRI along four dimensions—human, robot, task, and world and illustrates that previous HRI classifications can be successfully interpreted as either about one of these elements or about the relationship between two or more of these elements. Current HRI studies of single-operator single-robot (SOSR) control and single-operator multiple-robots (SOMR) control are reviewed using this approach. Human control of multiple robots has been suggested as a way to improve effectiveness in robot control. Unlike previous studies that investigated human interaction either in low-fidelity simulations or based on simple tasks, this thesis investigates human interaction with cooperating robot teams within a realistically complex environment. USARSim, a high-fidelity game-engine-based robot simulator, and MrCS, a distributed multirobot control system, were developed for this purpose. In the pilot experiment, we studied the impact of autonomy level. Mixed initiative control yielded performance superior to fully autonomous and manual control. To avoid limitation to particular application fields, the present thesis focuses on common HRI evaluations that enable us to analyze HRI effectiveness and guide HRI design independently of the robotic system or application domain. We introduce the interaction episode (IEP), which was inspired by our pilot human-multirobot control experiment, to extend the Neglect Tolerance model to support general multiple robots control for complex tasks. Cooperation Effort (CE), Cooperation Demand (CD), and Team Attention Demand (TAD) are defined to measure the cooperation in SOMR control. Two validation experiments were conducted to validate the CD measurement under tight and weak cooperation conditions in a high-fidelity virtual environment. The results show that CD, as a generic HRI metric, is able to account for the various factors that affect HRI and can be used in HRI evaluation and analysis.

[1]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[2]  Robin R. Murphy,et al.  Moonlight in Miami : A Field Study of Human-Robot Interaction in the Context of an Urban Search and Rescue Disaster Response Training Exercise , 2003 .

[3]  Luiz Chaimowicz,et al.  Deploying Air-Ground Multi-Robot Teams in Urban Environments , 2005 .

[4]  P. Tsang,et al.  Diagnosticity and multidimensional subjective workload ratings. , 1996, Ergonomics.

[5]  B. Trouvain,et al.  Evaluation of multi-robot control and monitoring performance , 2002, Proceedings. 11th IEEE International Workshop on Robot and Human Interactive Communication.

[6]  Jean Scholtz,et al.  Beyond usability evaluation: analysis of human-robot interaction at a major robotics competition , 2004 .

[7]  Mica R. Endsley,et al.  Automation and situation awareness. , 1996 .

[8]  Michael Lewis,et al.  Game Engine Virtual Reality with CaveUT , 2005, Computer.

[9]  Dan R. Olsen,et al.  Metrics for human driving of multiple robots , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[10]  Christopher M. Schlick,et al.  Comparison of a map- vs. camera-based user interface in a multi-robot navigation task , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[11]  Milind Tambe,et al.  Towards Flexible Teamwork , 1997, J. Artif. Intell. Res..

[12]  Jean Scholtz,et al.  Theory and evaluation of human robot interactions , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[13]  Milind Tambe,et al.  Allocating tasks in extreme teams , 2005, AAMAS '05.

[14]  Jean Scholtz,et al.  Common metrics for human-robot interaction , 2006, HRI '06.

[15]  Sara B. Kiesler,et al.  Fostering common ground in human-robot interaction , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[16]  Jean Scholtz,et al.  A field study of two techniques for situation awareness for robot navigation in urban search and rescue , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[17]  R. Murphy,et al.  Up from the Rubble: Lessons Learned about HRI from Search and Rescue , 2005 .

[18]  Robert W. Lindeman,et al.  Effectiveness of directional vibrotactile cuing on a building-clearing task , 2005, CHI.

[19]  M. Endsley.  Automation and situation awareness. , 1996 .

[20]  Thomas E. Nygren,et al.  The Subjective Workload Assessment Technique: A Scaling Procedure for Measuring Mental Workload , 1988 .

[21]  Dan R. Olsen,et al.  Metrics for Evaluating Human-Robot Interactions , 2003 .

[22]  Yehuda Elmaliach,et al.  Experiments with an ecological interface for monitoring tightly-coordinated robot teams , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[23]  Carl Gutwin,et al.  Workspace awareness for groupware , 1996, CHI Conference Companion.

[24]  Michael Lewis,et al.  Gravity-Referenced Attitude Display for Teleoperation of Mobile Robots , 2004 .

[25]  Kjeld Schmidt,et al.  The Problem with `Awareness': Introductory Remarks on `Awareness in CSCW' , 2002, Computer Supported Cooperative Work (CSCW).

[26]  Mica R. Endsley,et al.  Toward a Theory of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[27]  Richard T. Vaughan,et al.  The Player/Stage Project: Tools for Multi-Robot and Distributed Sensor Systems , 2003 .

[28]  Christopher A. Miller,et al.  Trust and etiquette in high-criticality automated systems , 2004, CACM.

[29]  J. Gregory Trafton,et al.  Enabling effective human-robot interaction using perspective-taking in robots , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[30]  Jean Scholtz,et al.  Evaluation of human-robot interaction awareness in search and rescue , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[31]  Terrence Fong,et al.  Sensor Fusion Based User Interface for Vehicle Teleoperation , 1999 .

[32]  N. Meiran,et al.  Component Processes in Task Switching , 2000, Cognitive Psychology.

[33]  Thomas D. Hardiman,et al.  A Comparative Analysis of Sagat and Sart for Evaluations of Situation Awareness , 1998 .

[34]  Raja Parasuraman,et al.  Beyond Levels of Automation: An Architecture for More Flexible Human-Automation Collaboration , 2003 .

[35]  Kara A. Latorella,et al.  The Scope and Importance of Human Interruption in Human-Computer Interaction Design , 2002, Hum. Comput. Interact..

[36]  Carl Gutwin,et al.  A Descriptive Framework of Workspace Awareness for Real-Time Groupware , 2002, Computer Supported Cooperative Work (CSCW).

[37]  John P. Lewis,et al.  The Future of Disaster Response: Humans Working with Multiagent Teams using DEFACTO , 2005, AAAI Spring Symposium: AI Technologies for Homeland Security.

[38]  Manuela M. Veloso,et al.  GameBots: a flexible test bed for multiagent team research , 2002, CACM.

[39]  Michael Jenkin,et al.  A Taxonomy of Multirobot Systems , 2003 .

[40]  John D. Lee,et al.  Trust in Automation: Designing for Appropriate Reliance , 2004, Hum. Factors.

[41]  Christopher D. Wickens,et al.  A model for types and levels of human interaction with automation , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[42]  Tom Rodden,et al.  Populating the application: a model of awareness for cooperative applications , 1996, CSCW '96.

[43]  C McFarlaneDaniel,et al.  The scope and importance of human interruption in human-computer interaction design , 2002 .

[44]  Jijun Wang.  Cooperating Robots for Search and Rescue , 2006 .

[45]  Stefano Carpin,et al.  Validating USARsim for use in HRI Research , 2005 .

[46]  Jean Scholtz,et al.  Awareness in human-robot interactions , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[47]  J. Gregory Trafton,et al.  Human control of multiple unmanned vehicles: effects of interface type on execution and task switching times , 2006, HRI '06.

[48]  Terrence Fong,et al.  Collaboration, Dialogue, and Human-Robot Interaction , 2001 .

[49]  R.R. Murphy,et al.  Human-robot interaction in USAR technical search: two heads are better than one , 2004, RO-MAN 2004. 13th IEEE International Workshop on Robot and Human Interactive Communication (IEEE Catalog No.04TH8759).

[50]  N. Moray,et al.  Adaptive automation, trust, and self-confidence in fault management of time-critical tasks. , 2000, Journal of experimental psychology. Applied.

[51]  Michael A. Goodrich,et al.  Validating human-robot interaction schemes in multitasking environments , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[52]  Stefania Bandini,et al.  Integrating Awareness in Cooperative Applications through the Reaction-Diffusion Metaphor , 2002, Computer Supported Cooperative Work (CSCW).

[53]  Holly A. Yanco,et al.  "Where am I?" Acquiring situation awareness using a remote robot platform , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[54]  Holly A. Yanco,et al.  A Taxonomy for Human-Robot Interaction , 2002 .

[55]  Robin R. Murphy,et al.  Human-robot interactions during the robot-assisted urban search and rescue response at the World Trade Center , 2003, IEEE Trans. Syst. Man Cybern. Part B.

[56]  Raja Parasuraman,et al.  The Effects of Level of Automation on the Out-of-the-Loop Unfamiliarity in a Complex Dynamic Fault-Management Task during Simulated Spaceflight Operations , 2001 .

[57]  Julie A. Adams,et al.  Visualization of Multiple Robots during Team Activities , 2006 .

[58]  Ronald L. Boring,et al.  Shared understanding for collaborative control , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[59]  Kurt Konolige,et al.  The saphira architecture for autonomous mobile robots , 1998 .

[60]  Illah R. Nourbakhsh,et al.  The role of expressiveness and attention in human-robot interaction , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[61]  Hiroshi Furukawa,et al.  A flexible delegation-type interface enhances system performance in human supervision of multiple robots: empirical studies with RoboFlag , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[62]  Jessica K. Hodgins,et al.  Modeling Physical Variability for Synthetic MOUT Agents , 2004 .

[63]  Julie A. Adams,et al.  Critical Considerations for Human-Robot Interface Development , 2002 .

[64]  Terrence Fong,et al.  Collaboration, Dialogue, Human-Robot Interaction , 2001, ISRR.

[65]  Mary L. Cummings,et al.  Managing Multiple UAVs through a Timeline Display , 2005 .

[66]  Jacob W. Crandall,et al.  Predicting Operator Capacity for Supervisory Control of Multiple UAVs , 2007, Innovations in Intelligent Machines.

[67]  Fumitoshi Matsuno,et al.  Time Follower's Vision: a teleoperation interface with past images , 2005, IEEE Computer Graphics and Applications.

[68]  Michael Lewis,et al.  Task-driven camera operations for robotic exploration , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[69]  Adam Jacoff,et al.  Experiences in Deploying Test Arenas for Autonomous Mobile Robots , 2001 .

[70]  Terrence Fong,et al.  Vehicle Teleoperation Interfaces , 2001, Auton. Robots.

[71]  Maja J. Mataric,et al.  New Directions: Robotics: Coordination and Learning in Multirobot Systems , 1998, IEEE Intell. Syst..

[72]  Brian Magerko,et al.  AI Characters and Directors for Interactive Computer Games , 2004, AAAI.

[73]  Bruce P. Hunn.  The Human Challenges of Command and Control with Multiple Unmanned Aerial Vehicles , 2005 .

[74]  John D. Lee,et al.  Trust in Automation: Designing for Appropriate Reliance , 2004 .

[75]  Brian Goldiez,et al.  A Survey of Commercial & Open Source Unmanned Vehicle Simulators , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[76]  Daniele Nardi,et al.  Multirobot systems: a classification focused on coordination , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[77]  Susana Rubio,et al.  Evaluation of Subjective Mental Workload: A Comparison of SWAT, NASA‐TLX, and Workload Profile Methods , 2004 .

[78]  Michael Lewis,et al.  Experiments with attitude: attitude displays for teleoperation , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[79]  Douglas E. McGovern,et al.  Experience and results in teleoperation of land vehicles , 1991 .

[80]  Lynne E. Parker,et al.  Lifelong Adaptation in Heterogeneous Multi-Robot Teams: Response to Continual Variation in Individual Robot Performance , 2000, Auton. Robots.

[81]  John L. Sibert,et al.  Interruption of people in human-computer interaction , 1998 .

[82]  Michael A. Goodrich,et al.  Towards facilitating the use of a pan-tilt camera on a mobile robot , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[83]  Michael A. Goodrich,et al.  Comparing the usefulness of video and map information in navigation tasks , 2006, HRI '06.

[84]  Stefano Carpin,et al.  Quantitative Assessments of USARSim Accuracy , 2006 .

[85]  Jean Scholtz,et al.  Evaluation of a human-robot interface: development of a situational awareness methodology , 2004, 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the.

[86]  Clifford Nass,et al.  Etiquette equality: exhibitions and expectations of computer politeness , 2004, CACM.

[87]  J. Sibert,et al.  Vibrotactile Feedback for Enhanced Control of Urban Search and Rescue Robots , 2007 .

[88]  Julie A. Adams,et al.  Task lists for human-multiple robot interaction , 2005, ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005..

[89]  Erika Rogers.  Human-Robot Interaction , 2004 .

[90]  S. Greenberg,et al.  The Importance of Awareness for Team Cognition in Distributed Collaboration , 2001 .

[91]  Oussama Khatib,et al.  Dynamic simulation of interactive robotic environment , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[92]  Mica R. Endsley,et al.  Design and Evaluation for Situation Awareness Enhancement , 1988 .

[93]  Michael A. Goodrich,et al.  Characterizing efficiency of human robot interaction: a case study of shared-control teleoperation , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[94]  Adam Jacoff,et al.  Traversability Metrics for Urban Search and Rescue Robots on Rough Terrain | NIST , 2007 .

[95]  David D. Woods,et al.  Envisioning human-robot coordination in future operations , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[96]  David B. Kaber,et al.  The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task , 2004 .

[97]  Andreas Birk,et al.  High Fidelity Tools for Rescue Robotics: Results and Perspectives , 2005, RoboCup.

[98]  Arnoud Visser,et al.  Towards heterogeneous robot teams for disaster mitigation: Results and performance metrics from RoboCup rescue , 2007, J. Field Robotics.

[99]  Marco Fratarcangeli,et al.  A 3D Simulator of Multiple Legged Robots Based on USARSim , 2006, RoboCup.

[100]  James S. Albus,et al.  4-D/RCS reference model architecture for unmanned ground vehicles , 1999, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[101]  Dan R. Olsen,et al.  Fan-out: measuring human control of multiple robots , 2004, CHI.

[102]  Steven Skiena,et al.  Attention and Communication: Decision Scenarios for Teleoperating Robots , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[103]  Jean Scholtz,et al.  Applying CSCW and HCI Techniques to Human-Robot Interaction , 2006 .

[104]  Donald D. Dudenhoeffer,et al.  Evaluation of supervisory vs. peer-peer interaction with human-robot teams , 2004, 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the.

[105]  H. Yanco,et al.  Camera Placement and Multi-Camera Fusion for Remote Robot Operation , 2006 .

[106]  S S Stevens,et al.  HUMAN ENGINEERING FOR AN EFFECTIVE AIR-NAVIGATION AND TRAFFIC-CONTROL SYSTEM, AND APPENDIXES 1 THRU 3 , 1951 .

[107]  Holly A. Yanco,et al.  Improved interfaces for human-robot interaction in urban search and rescue , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[108]  Douglas A. Few,et al.  Lessons learned from usability tests with a collaborative cognitive workspace for human-robot teams , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[109]  Holly A. Yanco,et al.  Classifying human-robot interaction: an updated taxonomy , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[110]  Terrence Fong,et al.  Multi-robot remote driving with collaborative control , 2003, IEEE Trans. Ind. Electron..

[111]  Michael R. M. Jenkin,et al.  A taxonomy for multi-agent robotics , 1996, Auton. Robots.

[112]  Robin R. Murphy,et al.  Moonlight in Miami: a field study of human-robot interaction in the context of an urban search and rescue disaster response training exercise , 2004 .

[113]  Craig Boutilier,et al.  Planning, Learning and Coordination in Multiagent Decision Processes , 1996, TARK.