Human Performance Issues and User Interface Design for Teleoperated Robots

In the future, it will become more common for humans to team up with robotic systems to perform tasks that humans cannot realistically accomplish alone. Even for autonomous and semiautonomous systems, teleoperation will be an important default mode. However, teleoperation can be a challenging task because the operator is remotely located. As a result, the operator's situation awareness of the remote environment can be compromised and the mission effectiveness can suffer. This paper presents a detailed examination of more than 150 papers covering human performance issues and suggested mitigation solutions. The paper summarizes the performance decrements caused by video images bandwidth, time lags, frame rates, lack of proprioception, frame of reference, two-dimensional views, attention switches, and motion effects. Suggested solutions and their limitations include stereoscopic displays, synthetic overlay, multimodal interfaces, and various predicative and decision support systems.

[1]  Paula J. Durlach,et al.  Human–Robot Interaction in the Context of Simulated Route Reconnaissance Missions , 2008 .

[2]  Stephen M. Casner,et al.  The Effects of GPS and Moving Map Displays on Pilot Navigational Awareness While Flying Under VFR , 2007 .

[3]  Patricia L. McDermott,et al.  Advantages of Co-Location for Effective Human to Human Communication of Information Provided by an Unmanned Vehicle , 2006 .

[4]  Jennifer E. Thropp,et al.  The Effects of Slow Frame Rates on Human Performance , 2006 .

[5]  Mark H. Draper,et al.  11. Multi-Sensory Interfaces for Remotely Operated Vehicles , 2006 .

[6]  Patricia L. McDermott,et al.  An investigation of real world control of robotic assets under communication latency , 2006, HRI '06.

[7]  Holly A. Yanco,et al.  Changing shape: improving situation awareness for a polymorphic robot , 2006, HRI '06.

[8]  Jean Scholtz,et al.  Common metrics for human-robot interaction , 2006, HRI '06.

[9]  Michael A. Goodrich,et al.  Comparing the usefulness of video and map information in navigation tasks , 2006, HRI '06.

[10]  B. A. MacDonald,et al.  Developer Oriented Visualisation of a Robot Program An Augmented Reality Approach , 2006 .

[11]  Sidney Fels,et al.  An Evaluation of a Vibro-Tactile Display Prototype for Physiological Monitoring , 2005, Anesthesia and analgesia.

[12]  Masatsugu Kidode,et al.  Robot Navigation by Eye Pointing , 2005, ICEC.

[13]  David Woods,et al.  Overcoming the Keyhole in Human-Robot Coordination: Simulation and Evaluation , 2005 .

[14]  W. T. Nelson,et al.  Target Acquisition With UAVs: Vigilance Displays and Advanced Cuing Interfaces , 2005, Hum. Factors.

[15]  R. Murphy,et al.  Up from the Rubble: Lessons Learned about HRI from Search and Rescue , 2005 .

[16]  Michael Lewis,et al.  Task-driven camera operations for robotic exploration , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[17]  Robert C. Anderson,et al.  The (in)accuracy of novice rover operators' perception of obstacle height from monoscopic images , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[18]  Stefano Caselli,et al.  Evaluation of virtual fixtures for a robot programming by demonstration interface , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[19]  Sergej Fatikow,et al.  Development of mobile versatile nanohandling microrobots: design, driving principles, haptic control , 2005, Robotica.

[20]  Robin R. Murphy,et al.  How UGVs physically fail in the field , 2005, IEEE Transactions on Robotics.

[21]  Susan G. Hill,et al.  Soldier Performance Issues in C2 `On the Move' , 2005 .

[22]  V. Grayson CuQlock-Knopp,et al.  Perception of terrain drop-offs as a function of L-R viewpoint separation in stereoscopic video , 2005, SPIE Defense + Commercial Sensing.

[23]  D. Yuh,et al.  Effect of sensory substitution on suture-manipulation forces for robotic surgical systems. , 2005, The Journal of thoracic and cardiovascular surgery.

[24]  Allison M. Okamura,et al.  Haptic Virtual Fixtures for Robot-Assisted Manipulation , 2005, ISRR.

[25]  Christopher D. Wickens,et al.  ATTENTIONAL TUNNELING AND TASK MANAGEMENT , 2005 .

[26]  Deb Roy,et al.  Elvis: situated speech and gesture understanding for a robotic chandelier , 2004, ICMI '04.

[27]  Hande Kaymaz-Keskinpala,et al.  Objective data analysis for a PDA-based human robotic interface , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[28]  Holly A. Yanco,et al.  Improved interfaces for human-robot interaction in urban search and rescue , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[29]  Michael A. Goodrich,et al.  Ecological displays for robot interaction: a new perspective , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[30]  Heath A. Ruff,et al.  Tactile versus Aural Redundant Alert Cues for UAV Control Applications , 2004 .

[31]  Michael I. Hill,et al.  Generalizeability of Latency Detection in a Variety of Virtual Environments , 2004 .

[32]  Michael Lewis,et al.  Gravity-Referenced Attitude Display for Teleoperation of Mobile Robots , 2004 .

[33]  Robin R. Murphy,et al.  Human-robot interaction in rescue robotics , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[34]  David D. Woods,et al.  Envisioning human-robot coordination in future operations , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[35]  Robert O. Ambrose,et al.  Telepresence control of the NASA/DARPA robonaut on a mobility platform , 2004, CHI.

[36]  A. Klein,et al.  Characterization of delay-induced piloting instability for the Triton undersea robot , 2004, 2004 IEEE Aerospace Conference Proceedings (IEEE Cat. No.04TH8720).

[37]  V. Zue.  Eighty Challenges Facing Speech Input/Output Technologies , 2004 .

[38]  A. Malcolm,et al.  Teleoperation Control for a Heavy-Duty Tracked Vehicle , 2004 .

[39]  Peter Xiaoping Liu,et al.  Visual gesture recognition for human-machine interface of robot teleoperation , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[40]  Pieter Padmos,et al.  Image parameters for driving with indirect viewing systems , 2003, Ergonomics.

[41]  Alex M. Andrew,et al.  Humans and Automation: System Design and Research Issues , 2003 .

[42]  Michael Lewis,et al.  Experiments with attitude: attitude displays for teleoperation , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[43]  Douglas A. Few,et al.  Lessons learned from usability tests with a collaborative cognitive workspace for human-robot teams , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[44]  Ali Safak Sekmen,et al.  Human robot interaction via cellular phones , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[45]  Joan Aranda,et al.  Visual human machine interface by gestures , 2003, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422).

[46]  J. Gregory Trafton,et al.  Finding the FOO: a pilot study for a multimodal interface , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[47]  Kazuhiko Kawamura,et al.  Evaluation of an enhanced human-robot interface , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[48]  Martin Jägersand,et al.  Predictive display system for tele-manipulation using image-based modeling and rendering , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[49]  Heath A. Ruff,et al.  Manual Versus Speech Input for Unmanned Aerial Vehicle Control Station Operations , 2003 .

[50]  Salvatore P Schipani,et al.  An Evaluation of Operator Workload, During Partially-Autonomous Vehicle Operations , 2003 .

[51]  Robin R. Murphy,et al.  Human-robot interactions during the robot-assisted urban search and rescue response at the World Trade Center , 2003, IEEE Trans. Syst. Man Cybern. Part B.

[52]  Imad H. Elhajj,et al.  Supermedia-enhanced Internet-based telerobotics , 2003, Proc. IEEE.

[53]  Pradeep K. Khosla,et al.  Psychophysical Characterization and Testbed Validation of a Wearable Vibrotactile Glove for Telemanipulation , 2003, Presence: Teleoperators & Virtual Environments.

[54]  David W. Payton,et al.  World embedded interfaces for human-robot interaction , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[55]  Masami Kobayashi,et al.  Telexistence cockpit for humanoid robot control , 2003, Adv. Robotics.

[56]  Charles E. Thorpe,et al.  PdaDriver: A Handheld System for Remote Driving , 2003 .

[57]  Marlin L. Gendron,et al.  Demonstration of a moving-map system for improved precise lane navigation of amphibious vehicles and landing craft , 2003, Oceans 2003. Celebrating the Past ... Teaming Toward the Future (IEEE Cat. No.03CH37492).

[58]  Thomas B. Sheridan,et al.  Telerobotics, Automation, and Human Supervisory Control , 2003 .

[59]  Gregory Hager,et al.  Human-Machine Collaborative Systems for Microsurgical Applications , 2003, ISRR.

[60]  Dan R. Olsen,et al.  Metrics for Evaluating Human-Robot Interactions , 2003 .

[61]  Yoshinori Kuno,et al.  Mutual assistance between speech and vision for human-robot interface , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[62]  J. Tittle,et al.  The Remote Perception Problem , 2002 .

[63]  Heath A. Ruff,et al.  Utilty of a Tactile Display for Cueing Faults , 2002 .

[64]  Craig R. Carignan,et al.  Effects of time delay on telerobotic control of neutral buoyancy vehicles , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[65]  Corinna E. Lathan,et al.  The Effects of Operator Spatial Perception and Sensory Feedback on Human-Robot Teleoperation Performance , 2002, Presence: Teleoperators & Virtual Environments.

[66]  Kunikatsu Takase,et al.  Multioperator Teleoperation of Multirobot Systems with Time Delay: Part IITestbed Description , 2002, Presence: Teleoperators & Virtual Environments.

[67]  Jaydev P. Desai,et al.  A Novel Approach to Robotic Cardiac Surgery Using Haptics and Vision , 2002 .

[68]  H. Sasaki.  Multioperator Teleoperation of Multirobot Systems with Time Delay: Part II—Testbed Description , 2002 .

[69]  Steffen Werner.  Cognitive Reference Systems and Their Role in Designing Spatial Information Displays , 2002, Künstliche Intell..

[70]  Monica Reggiani,et al.  A multimodal user interface for remote object exploration in teleoperation systems , 2002 .

[71]  Yoshinori Kuno,et al.  Human-robot interface based on the mutual assistance between speech and vision , 2001, PUI '01.

[72]  Daniel P. Siewiorek,et al.  Design of a wearable tactile display , 2001, Proceedings Fifth International Symposium on Wearable Computers.

[73]  Tianmiao Wang,et al.  The design of multimodal human-machine interface for teleoperation , 2001, 2001 IEEE International Conference on Systems, Man and Cybernetics. e-Systems and e-Man for Cybernetics in Cyberspace (Cat.No.01CH37236).

[74]  Jan B.F. van Erp,et al.  Driving with a Head-Slaved Camera System , 2001 .

[75]  Lisa C. Thomas,et al.  Visual Displays and Cognitive Tunneling: Frames of Reference Effects on Spatial Judgments and Change Detection , 2001 .

[76]  Charles E. Thorpe,et al.  Multi-robot remote driving with collaborative control , 2001, Proceedings 10th IEEE International Workshop on Robot and Human Interactive Communication. ROMAN 2001 (Cat. No.01TH8591).

[77]  Maria Bualat,et al.  Virtual Reality Interfaces for Visualization and Control of Remote Vehicles , 2001, Auton. Robots.

[78]  Terrence Fong,et al.  Vehicle Teleoperation Interfaces , 2001, Auton. Robots.

[79]  W B Toscano,et al.  Effects of Command and Control Vehicle (C2V) operational environment on soldier health and performance. , 1999, Human performance in extreme environments : the journal of the Society for Human Performance in Extreme Environments.

[80]  Barry Peterson,et al.  Effects of streaming video quality of service on spatial comprehension in a reconnaissance task , 2001 .

[81]  Lisa C. Thomas,et al.  Effects of Display Frames of Reference on Spatial Judgments and Change Detection , 2000 .

[82]  R W Cholewiak,et al.  The generation of vibrotactile patterns on a linear array: Influences of body site, time, and presentation mode , 2000, Perception & psychophysics.

[83]  Christopher D. Wickens,et al.  Tactical Displays for Combat Awareness: An Examination of Dimensionality and Frame of Reference Concepts and the Application of Cognitive Engineering , 2000 .

[84]  David B. Kaber,et al.  Effects of Visual Interface Design, and Control Mode and Latency on Performance, Telepresence and Workload in a Teleoperation Task , 2000 .

[85]  Christopher C. Smyth,et al.  Indirect Vision Driving with Fixed Flat Panel Displays for Near Unity, Wide, and Extended Fields of Camera View , 2000 .

[86]  D J Newman,et al.  A tactile display for international space station (ISS) extravehicular activity (EVA). , 2000, Aviation, space, and environmental medicine.

[87]  Edoardo Ardizzone,et al.  An architecture for automatic gesture analysis , 2000, AVI '00.

[88]  H. Rininsland.  ARTEMIS. A telemanipulator for cardiac surgery. , 1999, European journal of cardio-thoracic surgery : official journal of the European Association for Cardio-thoracic Surgery.

[89]  Yoshiaki Shirai,et al.  Robotic wheelchair with three control modes , 1999, Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C).

[90]  Rudy Darken,et al.  Map usage in virtual environments: orientation issues , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[91]  Bernard D. Adelstein,et al.  Sensor spatial distortion, visual latency, and update rate effects on 3D tracking in virtual environments , 1999, Proceedings IEEE Virtual Reality (Cat. No. 99CB36316).

[92]  David B. Kaber,et al.  Speculations on the Value of Telepresence , 1999, Cyberpsychology Behav. Soc. Netw..

[93]  Hideki Hashimoto,et al.  Teleoperated Nano Scale Object Manipulation , 1999 .

[94]  David B. Kaber,et al.  Telepresence , 1998, Hum. Factors.

[95]  Wallace J. Sadowski,et al.  Nonvisually Guided Locomotion to a Previously Viewed Target in Real and Virtual Environments , 1998, Hum. Factors.

[96]  William Ribarsky,et al.  Effects of Variation in System Responsiveness on User Performance in Virtual Environments , 1998, Hum. Factors.

[97]  Ronald R. Mourant,et al.  Human Factors Issues in Virtual Environments: A Review of the Literature , 1998, Presence.

[98]  Rafael Aracil,et al.  Telerobotic system for live power lines maintenance: ROBTET , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[99]  Debra J. Patton,et al.  Quantification of Cognitive Process Degradation While Mobile, Attributable to the Environmental Stressors Endurance, Vibration, and Noise , 1998 .

[100]  Bob G. Witmer,et al.  Judging Perceived and Traversed Distance in Virtual Environments , 1998, Presence.

[101]  David R. Scribner,et al.  The Effect of Stereoscopic and Wide Field of View Conditions on Teleoperator Performance , 1998 .

[102]  Cher Carney,et al.  Human Factors Design Guidelines for Advanced Traveler Information Systems (ATIS) and Commercial Vehicle Operations (CVO) , 1998 .

[103]  D. Simons,et al.  Failure to detect changes to attended objects in motion pictures , 1997 .

[104]  H. Das,et al.  Dexterity-enhanced telerobotic microsurgery , 1997, 1997 8th International Conference on Advanced Robotics. Proceedings. ICAR'97.

[105]  Daniel R. Tufano,et al.  Automotive HUDs: The Overlooked Safety Issues , 1997, Hum. Factors.

[106]  Joe Tullio,et al.  An integrated remote neurosurgical system , 1997, CVRMed.

[107]  William Ribarsky,et al.  Evaluation of the effects of frame time variation on VR task performance , 1997, Proceedings of IEEE 1997 Annual International Symposium on Virtual Reality.

[108]  Paul Richard,et al.  Multimodal virtual reality: Input-output devices, system integration, and human factors , 1996, Int. J. Hum. Comput. Interact..

[109]  Daniel E. Koditschek,et al.  Dynamical system representation, generation, and recognition of basic oscillatory motion gestures , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[110]  Paul Richard,et al.  Effect of Frame Rate and Force Feedback on Virtual Object Manipulation , 1996, Presence: Teleoperators & Virtual Environments.

[111]  James P. Bliss,et al.  Distance Estimation in Virtual Environments , 1995 .

[112]  Eugenia M. Kolasinski,et al.  Simulator Sickness in Virtual Environments. , 1995 .

[113]  Louis B. Rosenberg.  “Virtual fixtures”: perceptual overlays enhance operator performance in telepresence tasks , 1994 .

[114]  David N. Lee,et al.  Where we look when we steer , 1994, Nature.

[115]  Tracy H. Pastore.  Improved Operator Awareness of Teleoperated Land Vehicle Attitude. , 1994 .

[116]  T B Sheridan,et al.  Teleoperator Performance with Varying Force and Visual Feedback , 1994, Human factors.

[117]  Richard P. Paul,et al.  An Operator Interface for Teleprogramming Employing Synthetic Fixtures , 1994, Presence: Teleoperators & Virtual Environments.

[118]  David Drascic,et al.  Using stereoscopic video for defense teleoperation , 1993, Electronic Imaging.

[119]  Louis B. Rosenberg.  The effect of interocular distance upon operator performance using stereoscopic displays to perform virtual depth tasks , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.

[120]  Kellogg S. Booth,et al.  Evaluating 3D task performance for fish tank virtual worlds , 1993, TOIS.

[121]  I. Scott MacKenzie,et al.  Lag as a determinant of human performance in interactive systems , 1993, INTERCHI.

[122]  John V. Draper,et al.  Three experiments with stereoscopic television: when it works and why , 1991, Conference Proceedings 1991 IEEE International Conference on Systems, Man, and Cybernetics.

[123]  Douglas E. McGovern,et al.  Experience and results in teleoperation of land vehicles , 1991 .

[124]  Anthony J. Aretz.  The Design of Electronic Map Displays , 1991 .

[125]  Walter A. Aviles.  Issues in Mobile robotics: The Unmanned Ground Vehicle Program TeleOperated Vehicle , 1990 .

[126]  S. Kicha Ganapathy,et al.  A synthetic visual environment with hand gesturing and voice input , 1989, CHI '89.

[127]  W W Wierwille,et al.  Effects of Visual Display and Motion System Delays on Operator Performance and Uneasiness in a Driving Simulator , 1988, Human factors.

[128]  David D. Woods,et al.  Visual Momentum: A Concept to Improve the Cognitive Coupling of Person and Computer , 1984, Int. J. Man Mach. Stud..

[129]  R. Held,et al.  Adaptation to displaced and delayed visual feedback from the hand. , 1966 .

[130]  Thomas B. Sheridan,et al.  Remote Manipulative Control with Transmission Delay , 1963 .

[131]  P. Fitts.  The information capacity of the human motor system in controlling the amplitude of movement. , 1954, Journal of experimental psychology.