Projection-Aware Task Planning and Execution for Human-in-the-Loop Operation of Robots in a Mixed-Reality Workspace

Recent advances in mixed-reality technologies have renewed interest in alternative modes of communication for human-robot interaction. However, most of the work in this direction has been confined to tasks such as teleoperation, simulation or explication of individual actions of a robot. In this paper, we will discuss how the capability to project intentions affect the task planning capabilities of a robot. Specifically, we will start with a discussion on how projection actions can be used to reveal information regarding the future intentions of the robot at the time of task execution. We will then pose a new planning paradigm - projection-aware planning - whereby a robot can trade off its plan cost with its ability to reveal its intentions using its projection actions. We will demonstrate each of these scenarios with the help of a joint human-robot activity using the HoloLens.

[1]  Stephanie Rosenthal,et al.  Dynamic generation and refinement of robot verbalization , 2016, 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[2]  Daniel Bryce,et al.  A Tutorial on Planning Graph Based Reachability Heuristics , 2007, AI Mag..

[3]  Hector Geffner,et al.  Probabilistic Plan Recognition Using Off-the-Shelf Classical Planners , 2010, AAAI.

[4]  Rachel K. E. Bellamy,et al.  Visualizations for an Explainable Planning Agent , 2017, IJCAI.

[5]  Subbarao Kambhampati,et al.  Human-Aware Planning Revisited : A Tale of Three Models , 2018 .

[6]  Subbarao Kambhampati,et al.  Plan Explanations as Model Reconciliation - An Empirical Study , 2018, ArXiv.

[7]  Tathagata Chakraborti,et al.  Virtual, Augmented, and Mixed Reality for Human-Robot Interaction (VAM-HRI) , 2019, HRI.

[8]  Thomas B. Moeslund,et al.  Projecting robot intentions into human environments , 2016, 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[9]  Brian Scassellati,et al.  Autonomously constructing hierarchical task networks for planning and human-robot collaboration , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[10]  Jonathan P. How,et al.  MAR-CPS: Measurable Augmented Reality for Prototyping Cyber-Physical Systems , 2015 .

[11]  Ravi Teja Chadalavada,et al.  That's on my mind! robot to human intention communication through on-board projection on shared floor space , 2015, 2015 European Conference on Mobile Robots (ECMR).

[12]  Siddhartha S. Srinivasa,et al.  Effects of Robot Motion on Human-Robot Collaboration , 2015, 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[13]  Siddhartha S. Srinivasa,et al.  Generating Legible Motion , 2013, Robotics: Science and Systems.

[14]  Emanuele Ruffaldi,et al.  Third Point of View Augmented Reality for Robot Intentions Visualization , 2016, AVR.

[15]  Jonathan P. How,et al.  Measurable Augmented Reality for Prototyping Cyberphysical Systems: A Robotics Platform to Aid the Hardware Prototyping and Performance Testing of Algorithms , 2016, IEEE Control Systems.

[16]  Yu Zhang,et al.  Planning with Resource Conflicts in Human-Robot Cohabitation , 2016, AAMAS.

[17]  Yash K. Rathore Facilitating Human-Robot Collaboration Using a Mixed-Reality Projection System , 2017 .

[18]  Subbarao Kambhampati,et al.  Alternative Modes of Interaction in Proximal Human-in-the-Loop Operation of Robots , 2017, ArXiv.

[19]  Kentaro Ishii,et al.  Designing Laser Gesture Interface for Robot Control , 2009, INTERACT.

[20]  Rachel K. E. Bellamy,et al.  Mr. Jones - Towards a Proactive Smart Room Orchestrator , 2017, AAAI Fall Symposia.

[21]  Erez Karpas,et al.  Cost-Optimal Planning with Landmarks , 2009, IJCAI.

[22]  Atsushi Watanabe,et al.  Communicating robotic navigational intentions , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[23]  Subbarao Kambhampati,et al.  Handling Model Uncertainty and Multiplicity in Explanations via Model Reconciliation , 2018, ICAPS.

[24]  Shin Sato,et al.  A human-robot interface using an interactive hand pointer that projects a mark in the real work space , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[25]  Erez Karpas,et al.  Robust Execution of Plans for Human-Robot Teams , 2015, ICAPS.

[26]  Nancy M. Amato,et al.  A Roadmap for US Robotics - From Internet to Robotics 2020 Edition , 2021, Found. Trends Robotics.

[27]  Kaspar Althoefer,et al.  FourByThree: Imagine humans and robots working hand in hand , 2016, 2016 IEEE 21st International Conference on Emerging Technologies and Factory Automation (ETFA).

[28]  Sailik Sengupta,et al.  MA-RADAR – A Mixed-Reality Interface for Collaborative Decision Making , 2018 .

[29]  Nicholas R. Gans,et al.  A Multi-view camera-projector system for object detection and robot-human feedback , 2013, 2013 IEEE International Conference on Robotics and Automation.

[30]  Tom Brock,et al.  The Industrial Robot , 1983 .

[31]  Yu Zhang,et al.  Explicable Robot Planning as Minimizing Distance from Expected Behavior , 2016, ArXiv.

[32]  Kentaro Ishii,et al.  Blinkbot: look at, blink and move , 2010, UIST '10.

[33]  Ramsundar Kalpagam Ganesan Mediating Human-Robot Collaboration through Mixed Reality Cues , 2017 .

[34]  Nils J. Nilsson,et al.  A Formal Basis for the Heuristic Determination of Minimum Cost Paths , 1968, IEEE Trans. Syst. Sci. Cybern..

[35]  Klaus Schilling,et al.  A Spatial Augmented Reality system for intuitive display of robotic data , 2013, 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[36]  Miquel Ramírez,et al.  Action Selection for Transparent Planning , 2018, AAMAS.

[37]  Ross A. Knepper,et al.  Asking for Help Using Inverse Semantics , 2014, Robotics: Science and Systems.

[38]  Lukás Chrpa,et al.  Reformulating Planning Problems by Eliminating Unpromising Actions , 2009, SARA.

[39]  Yu Zhang,et al.  Plan explicability and predictability for robot task planning , 2015, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[40]  Gal A. Kaminka,et al.  Curing robot autism: a challenge , 2013, AAMAS.

[41]  Yu Zhang,et al.  Plan Explanations as Model Reconciliation: Moving Beyond Explanation as Soliloquy , 2017, IJCAI.

[42]  Yu Zhang,et al.  Planning for serendipity , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[43]  Subbarao Kambhampati,et al.  Balancing Explicability and Explanations - Emergent Behaviors in Human-Aware Planning , 2018, AAMAS 2018.

[44]  Matthew Turk,et al.  Computer Vision for Mobile Augmented Reality , 2015, Mobile Cloud Visual Media Computing.