Computational design of mixed-initiative human–robot teaming that considers human factors: situational awareness, workload, and workflow preferences

Advancements in robotic technology are making it increasingly possible to integrate robots into the human workspace in order to improve productivity and decrease worker strain resulting from the performance of repetitive, arduous physical tasks. While new computational methods have significantly enhanced the ability of people and robots to work flexibly together, there has been little study of the ways in which human factors influence the design of these computational techniques. In particular, collaboration with robots presents unique challenges related to the preservation of human situational awareness and the optimization of workload allocation for human teammates while respecting their workflow preferences. We conducted a series of human subject experiments to investigate these human factors, and provide design guidelines for the development of intelligent collaborative robots based on our results.

[1]  Piedmont Airlines,et al.  AIRCRAFT ACCIDENT REPORT , 1969 .

[2]  Peter Robinson,et al.  Challenges and Opportunities in Building Socially Intelligent Machines , 2011 .

[3]  Mary L. Cummings,et al.  Operator Performance and Intelligent Aiding in Unmanned Aerial Vehicle Scheduling , 2007, IEEE Intelligent Systems.

[4]  David B. Kaber,et al.  Design of Automation for Telerobots and the Effect on Performance, Operator Situation Awareness, and Subjective Workload , 2000 .

[5]  Julie A. Shah,et al.  Fast Scheduling of Multi-Robot Teams with Temporospatial Constraints , 2013, Robotics: Science and Systems.

[6]  David B. Kaber,et al.  Out‐of‐the‐loop performance problems and the use of intermediate levels of automation for improved control system functioning and safety , 1997 .

[7]  Edmund H. Durfee,et al.  Using hybrid scheduling for the semi-autonomous formation of expert teams , 2014, Future Gener. Comput. Syst..

[8]  Raja Parasuraman,et al.  Humans and Automation: Use, Misuse, Disuse, Abuse , 1997, Hum. Factors.

[9]  Leila Takayama,et al.  Influences on proxemic behaviors in human-robot interaction , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[10]  Martial Hebert,et al.  Autonomy Infused Teleoperation with Application to BCI Manipulation , 2015, Robotics: Science and Systems.

[11]  Penelope M. Sanderson,et al.  The Human Planning and Scheduling Role in Advanced Manufacturing Systems: An Emerging Human Factors Domain , 1989 .

[12]  Cameron H. Engh,et al.  Towards Using UAVs in Wilderness Search and Rescue: Lessons from Field Trials , 2009 .

[13]  Eric T. Chancey,et al.  Effects of Alarm Modality and Alarm Reliability on Workload, Trust, and Driving Performance , 2015 .

[14]  Liliana Ardissono,et al.  Mixed-Initiative Management of Online Calendars , 2012, WEBIST.

[15]  Ross A. Knepper,et al.  Asking for Help Using Inverse Semantics , 2014, Robotics: Science and Systems.

[16]  Meike Jipp,et al.  Effects of Observability, Mood States, and Workload on Human Handling Errors When Monitoring Aircraft Automation , 2012 .

[17]  Stefanos Nikolaidis,et al.  Human-robot cross-training: Computational formulation, modeling and evaluation of a human team training strategy , 2013, 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[18]  G. Glass,et al.  Consequences of Failure to Meet Assumptions Underlying the Fixed Effects Analyses of Variance and Covariance , 1972 .

[19]  Dimitris Bertsimas,et al.  Optimization over integers , 2005 .

[20]  MR Jones Autonomous Robots in SWAT Applications : Research , Design , and Operations Challenges , 2002 .

[21]  David B. Kaber,et al.  Situation awareness and attention allocation measures for quantifying telepresence experiences in teleoperation , 2004 .

[22]  Dieter Fox,et al.  Adapting the Sample Size in Particle Filters Through KLD-Sampling , 2003, Int. J. Robotics Res..

[23]  Wendy Ju,et al.  Animate Objects: How Physical Motion Encourages Public Interaction , 2010, PERSUASIVE.

[24]  Pamela S. Tsang,et al.  Mental Workload and Situation Awareness , 2006 .

[25]  Sean Hayes,et al.  A Comparison of Communicative Modes for Map-Based Tasking , 2011 .

[26]  Julie A. Shah,et al.  On Fairness in Decision-Making under Uncertainty: Definitions, Computation, and Comparison , 2015, AAAI.

[27]  Ho Chit Siu,et al.  Comparative Performance of Human and Mobile Robotic Assistants in Collaborative Fetch-and-Deliver Tasks , 2014, 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[28]  Siddhartha S. Srinivasa,et al.  Teleoperation with intelligent and customizable interfaces , 2013, Journal of Human-Robot Interaction.

[29]  Boi Faltings,et al.  A Multi-Agent Recommender System for Planning Meetings , 2000 .

[30]  Nicola Muscettola,et al.  Reformulating Temporal Plans for Efficient Execution , 1998, KR.

[31]  Il Hong Suh,et al.  A temporal Bayesian network with application to design of a proactive robotic assistant , 2012, 2012 IEEE International Conference on Robotics and Automation.

[32]  Tomás E. Uribe,et al.  Deploying a personalized time management agent , 2006, AAMAS '06.

[33]  Siddhartha S. Srinivasa,et al.  A policy-blending formalism for shared control , 2013, Int. J. Robotics Res..

[34]  N.A. Stanton,,et al.  Automatic Intelligent Cruise Control , 2006 .

[35]  Liliana Ardissono,et al.  Mixed-initiative Scheduling of Tasks in user Collaboration , 2012, WEBIST.

[36]  Gene V. Glass,et al.  Consequences of Failure to Meet Assumptions underlying the Analysis of Variance and Covariance. , 1972 .

[37]  Han-Lim Choi,et al.  Consensus-Based Auction Approaches for Decentralized Task Assignment , 2008 .

[38]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[39]  K. M. Lee,et al.  Can robots manifest personality? : An empirical test of personality recognition, social responses, and social presence in human-robot interaction , 2006 .

[40]  Ernesto Nunes,et al.  Multi-Robot Auctions for Allocation of Tasks with Temporal Constraints , 2015, AAAI.

[41]  Matthias Scheutz,et al.  Planning for human-robot teaming in open worlds , 2010, TIST.

[42]  Cynthia Breazeal,et al.  Effects of anticipatory action on human-robot teamwork: Efficiency, fluency, and perception of team , 2007, 2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[43]  James Carifio Measuring Vocational Preferences: Ranking versus Categorical Rating Procedures. , 1978 .

[44]  Kyle B. Reed,et al.  Physical Collaboration of Human-Human and Human-Robot Teams , 2008, IEEE Transactions on Haptics.

[45]  David B. Kaber,et al.  The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task , 2004 .

[46]  D. Medeiros,et al.  Accommodating individual preferences in nurse scheduling via auctions and optimization , 2009, Health Care Management Science.

[47]  Siddhartha S. Srinivasa,et al.  Assistive teleoperation of robot arms via automatic time-optimal mode switching , 2016, 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[48]  Cynthia Breazeal,et al.  Collaboration in Human-Robot Teams , 2004, AIAA 1st Intelligent Systems Technical Conference.

[49]  A. Horvath,et al.  Development and validation of the Working Alliance Inventory. , 1989 .

[50]  Terrence Fong,et al.  Vehicle Teleoperation Interfaces , 2001, Auton. Robots.

[51]  Mica R. Endsley,et al.  Toward a Theory of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[52]  Christopher D. Wickens,et al.  Multiple Resources and Mental Workload , 2008, Hum. Factors.

[53]  Guy Hoffman,et al.  Evaluating Fluency in Human–Robot Collaboration , 2019, IEEE Transactions on Human-Machine Systems.

[54]  Matthias Scheutz,et al.  Coordination in human-robot teams using mental modeling and plan recognition , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[55]  Scott R. Klemmer,et al.  How bodies matter: five themes for interaction design , 2006, DIS '06.

[56]  Kiran Lokhande,et al.  Cognitive Workload and Visual Attention Analyses of the Air Traffic Control Tower Flight Data Manager (TFDM) Prototype Demonstration , 2012 .

[57]  Ernesto Nunes,et al.  Exploiting Spatial Locality and Heterogeneity of Agents for Search and Rescue Teamwork * , 2016, J. Field Robotics.

[58]  Anthony Stentz,et al.  A comprehensive taxonomy for multi-robot task allocation , 2013, Int. J. Robotics Res..

[59]  Jean Scholtz,et al.  Common metrics for human-robot interaction , 2006, HRI '06.

[60]  R. M. Pierce,et al.  A data-driven method for determining natural human-robot motion mappings in teleoperation , 2012, 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[61]  Shayne Loft,et al.  Modeling and Predicting Mental Workload in En Route Air Traffic Control: Critical Review and Broader Implications , 2007, Hum. Factors.

[62]  Stefanos Nikolaidis,et al.  Optimization of Temporal Dynamics for Adaptive Human-Robot Interaction in Assembly Manufacturing , 2012, Robotics: Science and Systems.

[63]  Bilge Mutlu,et al.  Anticipatory robot control for efficient human-robot collaboration , 2016, 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[64]  Jonathan P. How,et al.  Operator Object Function Guidance for a Real-Time Unmanned Vehicle Scheduling Algorithm , 2012, J. Aerosp. Comput. Inf. Commun..

[65]  Sandip Sen,et al.  An automated meeting scheduling system that utilizes user preferences , 1997, AGENTS '97.

[66]  Minoru Asada,et al.  Initiative in robot assistance during collaborative task execution , 2016, 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[67]  Julie A. Shah,et al.  Decision-making authority, team efficiency and human worker satisfaction in mixed human–robot teams , 2015, Auton. Robots.

[68]  M. J. Soomer,et al.  Scheduling aircraft landings using airlines' preferences , 2008, Eur. J. Oper. Res..

[69]  George Mason Situation Awareness, Mental Workload, and Trust in Automation:Viable, Empirically Supported Cognitive Engineering Constructs , 2011 .

[70]  A. Kramer,et al.  Physiological metrics of mental workload: A review of recent progress , 1990, Multiple-task performance.

[71]  Anthony Stentz,et al.  Time-extended multi-robot coordination for domains with intra-path constraints , 2011, Auton. Robots.

[72]  Jessie Y. C. Chen,et al.  Human Performance Issues and User Interface Design for Teleoperated Robots , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[73]  Mica R. Endsley,et al.  Situation awareness global assessment technique (SAGAT) , 1988, Proceedings of the IEEE 1988 National Aerospace and Electronics Conference.

[74]  M R Endsley,et al.  Level of automation effects on performance, situation awareness and workload in a dynamic control task. , 1999, Ergonomics.

[75]  Florian Jentsch,et al.  Designing Effective Soldier-Robot Teams in Complex Environments: Training, Interfaces, and Individual Differences , 2011, HCI.

[76]  Peter Robinson,et al.  Challenges and Opportunities in Building Socially Intelligent Machines [Social Sciences] , 2011, IEEE Signal Processing Magazine.

[77]  Krzysztof Z. Gajos,et al.  Human computation tasks with global constraints , 2012, CHI.

[78]  Anca D. Dragan,et al.  Goal Inference Improves Objective and Perceived Performance in Human-Robot Collaboration , 2016, AAMAS.

[79]  Wendy Ju,et al.  Designing robots with movement in mind , 2014, Journal of Human-Robot Interaction.

[80]  Laurel D. Riek,et al.  A decomposition of UAV-related situation awareness , 2006, HRI '06.

[81]  Robert W. Proctor,et al.  Human factors in simple and complex systems , 1993 .

[82]  Glenn F. Wilson,et al.  Psychophysiological responses to changes in workload during simulated air traffic control , 1996, Biological Psychology.

[83]  Aaron F. Bobick,et al.  Anticipating human actions for collaboration in the presence of task and sensor uncertainty , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[84]  Siddhartha S. Srinivasa,et al.  Assitive teleoperation for manipulation tasks , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[85]  Sang Ryong Kim,et al.  Are physically embodied social agents better than disembodied social agents?: The effects of physical embodiment, tactile interaction, and people's loneliness in human-robot interaction , 2006, Int. J. Hum. Comput. Stud..

[86]  Ivan Lundberg,et al.  Safety of collaborative industrial robots: Certification possibilities for a collaborative assembly robot concept , 2011, 2011 IEEE International Symposium on Assembly and Manufacturing (ISAM).

[87]  J. A. Adams Multiple robot / single human interaction: effects on perceived workload , 2009, Behav. Inf. Technol..

[88]  Martin Buss,et al.  Human-Robot Collaboration: a Survey , 2008, Int. J. Humanoid Robotics.

[89]  Przemyslaw A. Lasota,et al.  Analyzing the Effects of Human-Aware Motion Planning on Close-Proximity Human–Robot Collaboration , 2015, Hum. Factors.

[90]  R. Chatila,et al.  Towards Human-Aware Cognitive Robots , 2006 .

[91]  James Carifio Scaling Preference Data for Program Assignments. , 1975 .

[92]  Terrence Fong,et al.  Collaboration, Dialogue, Human-Robot Interaction , 2001, ISRR.