Human-Robot Collaboration: A Literature Review and Augmented Reality Approach in Design

NASA's vision for space exploration stresses the cultivation of human-robotic systems. Similar systems are also envisaged for a variety of hazardous earthbound applications such as urban search and rescue. Recent research has pointed out that to reduce human workload, costs, fatigue driven error and risk, intelligent robotic systems will need to be a significant part of mission design. However, little attention has been paid to joint human-robot teams. Making human-robot collaboration natural and efficient is crucial. In particular, grounding, situational awareness, a common frame of reference and spatial referencing are vital in effective communication and collaboration. Augmented Reality (AR), the overlaying of computer graphics onto the real worldview, can provide the necessary means for a human-robotic system to fulfill these requirements for effective collaboration. This article reviews the field of human-robot interaction and augmented reality, investigates the potential avenues for creating natural human-robot collaboration through spatial dialogue utilizing AR and proposes a holistic architectural design for human-robot collaboration.

[1]  A. Kendon.  Some functions of gaze-direction in social interaction. , 1967, Acta psychologica.

[2]  V. Laxon,et al.  The conservation of number, mother, water and a fried egg chez l'enfant. , 1970, Acta psychologica.

[3]  H. H. Clark,et al.  Referring as a collaborative process , 1986, Cognition.

[4]  H. Clark,et al.  Grounding in Communication', 127-149 in Resnick LB, Levine JM and Teasley SD , 1991 .

[5]  Herbert H. Clark,et al.  Grounding in communication , 1991, Perspectives on socially shared cognition.

[6]  Stephanie D. Teasley,et al.  Perspectives on socially shared cognition , 1991 .

[7]  Shumin Zhai,et al.  Applications of augmented reality for human-robot communication , 1993, Proceedings of 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS '93).

[8]  Clifford Nass,et al.  Computers are social actors , 1994, CHI '94.

[9]  M. Studdert-Kennedy.  Hand and Mind: What Gestures Reveal About Thought. , 1994 .

[10]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[11]  Kenji Sakamoto,et al.  Multimodal Interaction in Human Communication , 1995, IEICE Trans. Inf. Syst..

[12]  Y. Inagaki,et al.  Behavior-based intention inference for intelligent robots cooperating with human , 1995, Proceedings of 1995 IEEE International Conference on Fuzzy Systems..

[13]  Lefteri H. Tsoukalas,et al.  Modeling instructible robots for waste disposal applications , 1996, Proceedings IEEE International Joint Symposia on Intelligence and Systems.

[14]  Mark Billinghurst,et al.  Wearable computers for three dimensional CSCW , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[15]  K. Suzuki,et al.  Development of a human and robot collaborative system for inspecting patrol of nuclear power plants , 1997, Proceedings 6th IEEE International Workshop on Robot and Human Communication. RO-MAN'97 SENDAI.

[16]  Mark Billinghurst,et al.  Spatial Information Displays on a Wearable Computer , 1998, IEEE Computer Graphics and Applications.

[17]  Koji Shibuya,et al.  Design and control of mobile manipulation system for human symbiotic humanoid: Hadaly-2 , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[18]  Paul U. Lee,et al.  Why do speakers mix perspectives? , 1999, Spatial Cogn. Comput..

[19]  K. Chang,et al.  Embodiment in conversational interfaces: Rea , 1999, CHI '99.

[20]  Kunikatsu Takase,et al.  Remote collaboration through time delay in multiple teleoperation , 1999, Proceedings 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human and Environment Friendly Robots with High Intelligence and Emotional Quotients (Cat. No.99CH36289).

[21]  Illah R. Nourbakhsh,et al.  An Affective Mobile Robot Educator with a Full-Time Job , 1999, Artif. Intell..

[22]  Steven K. Feiner,et al.  Information filtering for mobile augmented reality , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[23]  Tetsuo Sawaragi,et al.  Naturalistic human-robot collaboration based upon mixed-initiative interactions in teleoperating environment , 2000, Smc 2000 conference proceedings. 2000 ieee international conference on systems, man and cybernetics. 'cybernetics evolving to systems, humans, organizations, and their complex interactions' (cat. no.0.

[24]  Ivan Poupyrev,et al.  Mixing realities in Shared Space: an augmented reality interface for collaborative computing , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[25]  Ivan Poupyrev,et al.  The MagicBook: a transitional AR interface , 2001, Comput. Graph..

[26]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[27]  Nak Young Chong,et al.  Exploring interactive simulator in collaborative multi-site teleoperation , 2001, Proceedings 10th IEEE International Workshop on Robot and Human Interactive Communication. ROMAN 2001 (Cat. No.01TH8591).

[28]  Brian Scassellati,et al.  Active vision for sociable robots , 2001, IEEE Trans. Syst. Man Cybern. Part A.

[29]  Carlos Balaguer,et al.  Active human-mobile manipulator cooperation through intention recognition , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[30]  Yukiko I. Nakano,et al.  Non-Verbal Cues for Discourse Structure , 2022 .

[31]  Jean Scholtz,et al.  Human-Robot Interactions: Creating Synergistic Cyber Forces , 2002 .

[32]  Tetsuo Ono,et al.  Development and evaluation of an interactive humanoid robot "Robovie" , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[33]  Hirokazu Kato,et al.  Interactive Theatre Experience in Embodied + Wearable Mixed Reality Space , 2002, ISMAR.

[34]  Information Filtering for Mobile Augmented Reality , 2002, IEEE Computer Graphics and Applications.

[35]  Marjorie Skubic,et al.  Using spatial language in a human-robot dialog , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[36]  Anoop Gupta,et al.  Communication behaviors of co-located users in collaborative AR interfaces , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[37]  Kerstin Fischer,et al.  Spatial Strategies in Human-Robot Communication , 2002, Künstliche Intell..

[38]  Charles E. Thorpe,et al.  Robot as Partner: Vehicle Teleoperation with Collaborative Control , 2002 .

[39]  Matthias Denecke.  Rapid Prototyping for Spoken Dialogue Systems , 2002, COLING.

[40]  Hirokazu Kato,et al.  3-D live: real time interaction for mixed reality , 2002, CSCW '02.

[41]  James C. Maida,et al.  Utilization of the Space Vision System as an Augmented Reality System For Mission Operations , 2003 .

[42]  Wei Liu,et al.  Human Pacman: A Mobile Entertainment System with Ubiquitous Computing and Tangible Interaction over a Wide Outdoor Area , 2003, Mobile HCI.

[43]  Christian Faubel,et al.  Anthropomorphism as a pervasive design concept for a robotic assistant , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[44]  Philip R. Cohen,et al.  Referring as a Collaborative Process , 2003 .

[45]  Dieter Schmalstieg,et al.  Collaborative Augmented Reality for Outdoor Navigation and Information Browsing , 2003 .

[46]  Terrence Fong,et al.  Multi-robot remote driving with collaborative control , 2003, IEEE Trans. Ind. Electron..

[47]  Jean Scholtz,et al.  Theory and evaluation of human robot interactions , 2003, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[48]  Candace L. Sidner,et al.  Engagement Rules for Human-Robot Collaborative Interaction , 2003 .

[49]  Robert E. Kraut,et al.  Effects of head-mounted and scene-oriented video systems on remote collaboration on physical tasks , 2003, CHI '03.

[50]  Candace L. Sidner,et al.  Engagement rules for human-robot collaborative interactions , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[51]  Terrence Fong,et al.  Robot, asker of questions , 2003, Robotics Auton. Syst..

[52]  Yael Edan,et al.  Human-robot collaboration for improved target recognition of agricultural robots , 2003, Ind. Robot.

[53]  Hideaki Kuzuoka,et al.  Dual ecologies of robot as communication media: thoughts on coordinating orientations and projectability , 2004, CHI.

[54]  Cory D. Kidd,et al.  HUMANOID ROBOTS AS COOPERATIVE PARTNERS FOR PEOPLE , 2004 .

[55]  Anders Green,et al.  Involving users in the design of a mobile office robot , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[56]  Cynthia Breazeal,et al.  Social interactions in HRI: the robot view , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[57]  Rüdiger Dillmann,et al.  Using augmented reality to interact with an autonomous mobile platform , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[58]  Robert O. Ambrose,et al.  Cooperative manipulation between humans and teleoperated agents , 2004, 12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2004. HAPTICS '04. Proceedings..

[59]  Jean Scholtz,et al.  Beyond Usability Evaluation: Analysis of Human-Robot Interaction at a Major Robotics Competition , 2004, Hum. Comput. Interact..

[60]  Cynthia Breazeal,et al.  Robots that Work in Collaboration with People , 2004, AAAI Technical Report.

[61]  Robin R. Murphy,et al.  Human-robot interaction in rescue robotics , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[62]  Nilanjan Sarkar,et al.  Anxiety detecting robotic system – towards implicit human-robot collaboration , 2004, Robotica.

[63]  Jean Scholtz,et al.  Beyond usability evaluation: analysis of human-robot interaction at a major robotics competition , 2004 .

[64]  Marjorie Skubic,et al.  Spatial language for human-robot dialogs , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[65]  Deb Roy,et al.  Mental imagery for a conversational robot , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[66]  Ruediger Dillmann,et al.  Sharing skills: using augmented reality for human-robot collaboration , 2004, IS&T/SPIE Electronic Imaging.

[67]  Mark Billinghurst,et al.  Designing augmented reality interfaces , 2005, COMG.

[68]  Illah R. Nourbakhsh,et al.  Interaction challenges in human-robot space exploration , 2005, INTR.

[69]  Illah R. Nourbakhsh,et al.  Human-robot teaming for search and rescue , 2005, IEEE Pervasive Computing.

[70]  Candace L. Sidner,et al.  Robots as laboratory hosts , 2005, INTR.

[71]  James C. Maida,et al.  Enhanced Lighting Techniques and Augmented Reality to Improve Human Task Performance , 2005 .

[72]  Bruce A. MacDonald,et al.  Developer oriented visualisation of a robot program , 2006, HRI '06.

[73]  Henrik I. Christensen,et al.  Clarification dialogues in human-augmented mapping , 2006, HRI '06.

[74]  Terrence Fong,et al.  The human-robot interaction operating system , 2006, HRI '06.

[75]  Michael A. Goodrich,et al.  Comparing Situation Awareness for Two Unmanned Aerial Vehicle Human Interface Approaches , 2006 .

[76]  E. L. Harder,et al.  The Institute of Electrical and Electronics Engineers, Inc. , 2019, 2019 IEEE International Conference on Software Architecture Companion (ICSA-C).