Modeling humans as peers and supervisors in computing systems through runtime models

There is a growing demand for more effective integration of humans and computing systems, specifically in multiagent and multirobot systems. There are two aspects to consider in human integration: (1) the ability to control an arbitrary number of robots (particularly heterogeneous robots) and (2) integrating humans as peers in computing systems instead of being just users or supervisors. With traditional supervisory control of multirobot systems, the number of robots that a human can manage effectively is between four and six [17]. A limitation of traditional supervisory control is that the human must interact individually with each robot, which limits the upper-bound on the number of robots that a human can control effectively. In this work, I define the concept of "organizational control" together with an autonomous mechanism that can perform task allocation and other low-level housekeeping duties, which significantly reduces the need for the human to interact with individual robots. Humans are very versatile and robust in the types of tasks they can accomplish. However, failures in computing systems are common and thus redundancies are included to mitigate the chance of failure. When all redundancies have failed, system failure will occur and the computing system will be unable to accomplish its tasks. One way to further reduce the chance of a system failure is to integrate humans as peer "agents" in the computing system. As part of the system, humans can be assigned tasks that would have been impossible to complete due to failures.

[1]  Lovekesh Vig,et al.  Coalition Formation: From Software Agents to Robots , 2007, J. Intell. Robotic Syst..

[2]  Barry G. Silverman,et al.  Taxonomy and Method for Handling Large and Diverse Sets of Interactive Objects in Immersive Environments , 2010 .

[3]  Scott A. DeLoach,et al.  agentTool process editor: supporting the design of tailored agent-based processes , 2009, SAC '09.

[4]  Zhihua Qu,et al.  RoboLeader for reconnaissance by a team of robotic vehicles , 2010, 2010 International Symposium on Collaborative Technologies and Systems.

[5]  Barry G. Silverman,et al.  NonKin Village: An Embeddable Training Game Generator for Learning Cultural Terrain and Sustainable Counter-Insurgent Operations , 2009, AGS.

[6]  Scott A. DeLoach Modeling Organizational Rules in the Multi-agent Systems Engineering Methodology , 2002, Canadian Conference on AI.

[7]  Lynne E. Parker,et al.  ALLIANCE: an architecture for fault tolerant multirobot cooperation , 1998, IEEE Trans. Robotics Autom..

[8]  Sarvapali D. Ramchurn,et al.  Decentralized Coordination in RoboCup Rescue , 2010, Comput. J..

[9]  Fabio Gagliardi Cozman,et al.  Safeguarded Teleoperation for Lunar Rovers: From Human Factors to Field Trials , 1999 .

[10]  Scott A. DeLoach,et al.  A Goal Model for Adaptive Complex Systems , 2009 .

[11]  Matthew Miller,et al.  A GOAL MODEL FOR DYNAMIC SYSTEMS , 2007 .

[12]  Jeffrey D. Anderson,et al.  Managing autonomy in robot teams: Observations from four experiments , 2007, 2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[13]  P. Blau Formal Organization: Dimensions of Analysis , 1957, American Journal of Sociology.

[14]  Jean Scholtz,et al.  Theory and evaluation of human robot interactions , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[15]  Kimon P. Valavanis,et al.  Optimized task allocation in cooperative robot teams , 2009, 2009 17th Mediterranean Conference on Control and Automation.

[16]  Nicholas R. Jennings,et al.  Market-Based Task Allocation Mechanisms for Limited-Capacity Suppliers , 2007, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[17]  John M. Dolan,et al.  Scheduling for humans in multirobot supervisory control , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[18]  Christopher D. Wickens,et al.  A model for types and levels of human interaction with automation , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[19]  John S. Bay,et al.  Toward the development of a material transport system using swarms of ant-like robots , 1993, [1993] Proceedings IEEE International Conference on Robotics and Automation.

[20]  Scott A. DeLoach,et al.  O-MaSE: a customisable approach to designing and building complex, adaptive multi-agent systems , 2010, Int. J. Agent Oriented Softw. Eng..

[21]  Scott A. DeLoach,et al.  An Agent-Environment Interaction Model , 2006, AOSE.

[22]  Han-Lim Choi,et al.  Consensus-Based Decentralized Auctions for Robust Task Allocation , 2009, IEEE Transactions on Robotics.

[23]  Jane W.-S. Liu Real-Time Systems , 2000, Encyclopedia of Algorithms.

[24]  Anthony Stentz,et al.  Traderbots: a new paradigm for robust and efficient multirobot coordination in dynamic environments , 2004 .

[25]  Michael Lewis,et al.  Task-driven camera operations for robotic exploration , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[26]  Mica R. Endsley,et al.  Theoretical Underpinnings of Situation Awareness, A Critical Review , 2000 .

[27]  Scott A. DeLoach,et al.  A capabilities-based model for adaptive organizations , 2008, Autonomous Agents and Multi-Agent Systems.

[28]  Scott A. DeLoach,et al.  Multiagent Systems Engineering , 2001, Int. J. Softw. Eng. Knowl. Eng..

[29]  Ronald L. Boring,et al.  Shared understanding for collaborative control , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[30]  Gerhard Weiss,et al.  Computational Organization Theory , 2000 .

[31]  Robert J. McEliece,et al.  The generalized distributive law , 2000, IEEE Trans. Inf. Theory.

[32]  Yu Zhang,et al.  A general information quality based approach for satisfying sensor constraints in multirobot tasks , 2010, 2010 IEEE International Conference on Robotics and Automation.

[33]  Javier Vázquez-Salceda,et al.  OMNI: Introducing Social Structure, Norms and Ontologies into Agent Organizations , 2004, PROMAS.

[34]  Javier Vázquez-Salceda,et al.  Modelling Electronic Organizations , 2003, CEEMAS.

[35]  Robert P. Goldman,et al.  "Tasking" Interfaces for Flexible Interaction with Automa- tion: Keeping the Operator in Control , 1999 .

[36]  Ichiro Suzuki,et al.  Distributed algorithms for formation of geometric patterns with many mobile robots , 1996, J. Field Robotics.

[37]  Zsolt Kira,et al.  Exerting human control over decentralized robot swarms , 2000, 2009 4th International Conference on Autonomous Robots and Agents.

[38]  Michael A. Goodrich,et al.  Designing human-centered automation: trade-offs in collision avoidance system design , 2000, IEEE Trans. Intell. Transp. Syst..

[39]  Manuela M. Veloso,et al.  Task Decomposition, Dynamic Role Assignment, and Low-Bandwidth Communication for Real-Time Strategic Teamwork , 1999, Artif. Intell..

[40]  Yoshiyuki Tanaka,et al.  Tracking control properties of human-robotic systems based on impedance control , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[41]  Hong Zhang,et al.  Collective Robotics: From Social Insects to Robots , 1993, Adapt. Behav..

[42]  Andreas Birk,et al.  Augmented autonomy: Improving human-robot team performance in Urban search and rescue , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[43]  Douglas C. Schmidt,et al.  Guest Editor's Introduction: Model-Driven Engineering , 2006, Computer.

[44]  X.C. Ding,et al.  An optimal timing approach to controlling multiple UAVs , 2009, 2009 American Control Conference.

[45]  David D. Woods,et al.  Envisioning human-robot coordination in future operations , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[46]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[47]  Robert C. Anderson,et al.  The (in)accuracy of novice rover operators' perception of obstacle height from monoscopic images , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[48]  Lovekesh Vig,et al.  Multi-robot coalition formation , 2006, IEEE Transactions on Robotics.

[49]  Paul Scerri,et al.  Coordinating very large groups of wide area search munitions , 2004 .

[50]  Jean Scholtz,et al.  Implementation of a situation awareness assessment tool for evaluation of human-robot interfaces , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[51]  Julie A. Adams,et al.  Supporting Human Supervision of Multiple Robots , 2006 .

[52]  Terrence Fong,et al.  A Safeguarded Teleoperation Controller , 2001 .

[53]  David Woods,et al.  1. How to make automated systems team players , 2002 .

[54]  Jeffrey O. Kephart,et al.  The Vision of Autonomic Computing , 2003, Computer.

[55]  Franco Zambonelli,et al.  Organisational Rules as an Abstraction for the Analysis and Design of Multi-Agent Systems , 2001, Int. J. Softw. Eng. Knowl. Eng..

[56]  Kevin M. Passino,et al.  Biomimicry of bacterial foraging for distributed optimization and control , 2002 .

[57]  Axel van Lamsweerde,et al.  Managing Conflicts in Goal-Driven Requirements Engineering , 1998, IEEE Trans. Software Eng..

[58]  Tucker R. Balch,et al.  Communication in reactive multiagent robotic systems , 1995, Auton. Robots.

[59]  Masao Kubo,et al.  Learning coordinated motions in a competition for food between ant colonies , 1994 .

[60]  Michael A. Goodrich,et al.  Validating human-robot interaction schemes in multitasking environments , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[61]  Reid G. Simmons,et al.  First Results in the Coordination of Heterogeneous Robots for Large-Scale Assembly , 2000, ISER.

[62]  Jacob W. Crandall,et al.  Developing performance metrics for the supervisory control of multiple robots , 2007, 2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[63]  Maja J. Mataric,et al.  Sold!: auction methods for multirobot coordination , 2002, IEEE Trans. Robotics Autom..

[64]  James McLurkin,et al.  Distributed Algorithms for Dispersion in Indoor Environments Using a Swarm of Autonomous Mobile Robots , 2004, DARS.

[65]  Kuanhao Zheng,et al.  How many social robots can one operator control? , 2011, 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[66]  Shuzhi Sam Ge,et al.  COBOS: Cooperative backoff adaptive scheme for multirobot task allocation , 2005, IEEE Transactions on Robotics.

[67]  Maarten Sierhuis,et al.  Toward Coactivity , 2010, 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[68]  Yu Zhang,et al.  IQ-ASyMTRe: Synthesizing coalition formation and execution for tightly-coupled multirobot tasks , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[69]  J. Gregory Trafton,et al.  Enabling effective human-robot interaction using perspective-taking in robots , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[70]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[71]  Hiroshi Furukawa,et al.  A flexible delegation-type interface enhances system performance in human supervision of multiple robots: empirical studies with RoboFlag , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[72]  Scott A. DeLoach,et al.  O-MaSE: A Customizable Approach to Developing Multiagent Development Processes , 2007, AOSE.

[73]  Stephen Whitlow,et al.  A playbook interface for mixed initiative control of multiple unmanned vehicle teams , 2002, Proceedings. The 21st Digital Avionics Systems Conference.

[74]  Lynne E. Parker,et al.  Distributed multi-robot coalitions through ASyMTRe-D , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[75]  Raja Parasuraman,et al.  Designing for Flexible Interaction Between Humans and Automation: Delegation Interfaces for Supervisory Control , 2007, Hum. Factors.

[76]  R. D'Andrea,et al.  The RoboFlag testbed , 2003, Proceedings of the 2003 American Control Conference, 2003..

[77]  M. V. Dignum,et al.  A Model for Organizational Interaction: based on Agents, founded in Logic , 2000 .

[78]  Kwee-Bo Sim,et al.  Artificial immune-based swarm behaviors of distributed autonomous robotic systems , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[79]  Thomas Stützle,et al.  Ant colony optimization: artificial ants as a computational intelligence technique , 2006 .

[80]  Laurel Allender,et al.  Verification, Validation, and Accreditation of a Soldier-System Modeling Tool , 1995 .

[81]  R.C. Arkin Reactive control as a substrate for telerobotic systems , 1991, NTC '91 - National Telesystems Conference Proceedings.

[82]  Zhihua Qu,et al.  RoboLeader: an agent for supervisory control of multiple robots , 2010, HRI 2010.

[83]  James McLurkin,et al.  Speaking Swarmish: Human-Robot Interface Design for Large Swarms of Autonomous Mobile Robots , 2006, AAAI Spring Symposium: To Boldly Go Where No Human-Robot Team Has Gone Before.

[84]  Michael Lewis,et al.  Experiments with attitude: attitude displays for teleoperation , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[85]  Katia P. Sycara,et al.  Synchronous vs. Asynchronous Control for Large Robot Teams , 2011, HCI.

[86]  Gordon S. Blair,et al.  Models@ run.time , 2009, Computer.

[87]  N. Badler,et al.  Crowd simulation incorporating agent psychological models, roles and communication , 2005 .

[88]  Sarvapali D. Ramchurn,et al.  A Distributed Anytime Algorithm for Dynamic Task Allocation in Multi-Agent Systems , 2011, AAAI.

[89]  Yili Liu,et al.  Introduction to Human Factors Engineering (2nd Edition) , 2003 .

[90]  Kurt Konolige,et al.  Distributed multirobot exploration, mapping, and task allocation , 2008, Annals of Mathematics and Artificial Intelligence.

[91]  Lynne E. Parker,et al.  ASyMTRe: Automated Synthesis of Multi-Robot Task Solutions through Software Reconfiguration , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[92]  Shafiq Abedin,et al.  Scalable target detection for large robot teams , 2011, 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[93]  Barry G. Silverman,et al.  Toward Realism in Human Performance Simulation , 2004 .

[94]  Maja J. Matarić,et al.  Designing emergent behaviors: from local interactions to collective intelligence , 1993 .

[95]  Dominic Létourneau,et al.  Autonomous spherical mobile robot for child-development studies , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[96]  Lynne E. Parker,et al.  Distributed Intelligence: Overview of the Field and Its Application in Multi-Robot Systems , 2008, AAAI Fall Symposium: Regarding the Intelligence in Distributed Intelligent Systems.

[97]  Marvin Minsky,et al.  Polyscheme: a cognitive architecture for integrating multiple representation and inference schemes , 2002 .

[98]  Magnus Egerstedt,et al.  Executive Decision Support: Single-Agent Control of Multiple UAVs , 2009 .

[99]  Rachid Alami,et al.  M+: a scheme for multi-robot cooperation through negotiated task allocation and achievement , 1999, Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C).

[100]  Alberto Elfes,et al.  Multi-Level Autonomy Robot Telesupervision , 2008 .

[101]  Richard A. Volz,et al.  Teleautonomous systems: projecting and coordinating intelligent action at a distance , 1990, IEEE Trans. Robotics Autom..

[102]  Kevin O'Brien,et al.  Human Behavior Models for Agents in Simulators and Games: Part I: Enabling Science with PMFserv , 2006, Presence: Teleoperators & Virtual Environments.

[103]  Gaurav S. Sukhatme,et al.  Article in Press Robotics and Autonomous Systems ( ) – Robotics and Autonomous Systems Multi-robot Task Allocation through Vacancy Chain Scheduling , 2022 .