Situated Live Programming for Human-Robot Collaboration

We present situated live programming for human-robot collaboration, an approach that enables users with limited programming experience to program collaborative applications for human-robot interaction. Allowing end users, such as shop floor workers, to program collaborative robots themselves would make it easy to “retask” robots from one process to another, facilitating their adoption by small and medium enterprises. Our approach builds on the paradigm of trigger-action programming (TAP) by allowing end users to create rich interactions through simple trigger-action pairings. It enables end users to iteratively create, edit, and refine a reactive robot program while executing partial programs. This live programming approach enables the user to utilize the task space and objects by incrementally specifying situated trigger-action pairs, substantially lowering the barrier to entry for programming or reprogramming robots for collaboration. We instantiate situated live programming in an authoring system where users can create trigger-action programs by annotating an augmented video feed from the robot’s perspective and assign robot actions to trigger conditions. We evaluated this system in a study where participants (n = 10) developed robot programs for solving collaborative light-manufacturing tasks. Results showed that users with little programming experience were able to program HRC tasks in an interactive fashion and our situated live programming approach further supported individualized strategies and workflows. We conclude by discussing opportunities and limitations of the proposed approach, our system implementation, and our study and discuss a roadmap for expanding this approach to a broader range of tasks and applications.

[1]  Fabio Paternò,et al.  Trigger-Action Programming for Personalising Humanoid Robot Behaviour , 2019, CHI.

[2]  Bilge Mutlu,et al.  Corrective Shared Autonomy for Addressing Task Variability , 2021, IEEE Robotics and Automation Letters.

[3]  Steven L. Tanimoto,et al.  VIVA: A visual language for image processing , 1990, J. Vis. Lang. Comput..

[4]  Chien-Ming Huang,et al.  PATI: a projection-based augmented table-top interface for robot programming , 2019, IUI.

[5]  J. B. Brooke,et al.  SUS: a retrospective , 2013 .

[6]  Nick Collins,et al.  Live coding in laptop performance , 2003, Organised Sound.

[7]  Blase Ur,et al.  Practical trigger-action programming in the smart home , 2014, CHI.

[8]  Shan Lu,et al.  AutoTap: Synthesizing and Repairing Trigger-Action Programs Using LTL Properties , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE).

[9]  Clare Dixon,et al.  Formal Specification and Verification of Autonomous Robotic Systems , 2018, ACM Comput. Surv..

[10]  Pavel Smrz,et al.  Combining Interactive Spatial Augmented Reality with Head-Mounted Display for End-User Collaborative Robot Programming , 2019, 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN).

[11]  Sorin Lerner.  Focused Live Programming with Loop Seeds , 2020, UIST.

[12]  Shimon Y. Nof,et al.  Vipo: Spatial-Visual Programming with Functions for Robot-IoT Workflows , 2020, CHI.

[13]  Kris Hauser,et al.  In-hand Object Scanning via RGB-D Video Segmentation , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[14]  Bilge Mutlu,et al.  Collaborative or Simply Uncaged? Understanding Human-Cobot Interactions in Automation , 2020, CHI.

[15]  Bilge Mutlu,et al.  Authr: A Task Authoring Environment for Human-Robot Teams , 2020, UIST.

[16]  Freek Stulp,et al.  Intuitive Task-Level Programming by Demonstration Through Semantic Skill Recognition , 2019, IEEE Robotics and Automation Letters.

[17]  Jiyun Lee,et al.  Trigger-Action Programming in the Wild: An Analysis of 200,000 IFTTT Recipes , 2016, CHI.

[18]  Julie A. Shah,et al.  Fast Scheduling of Robot Teams Performing Tasks With Temporospatial Constraints , 2018, IEEE Transactions on Robotics.

[19]  Alexandre Bergel,et al.  The Road to Live Programming: Insights from the Practice , 2018, 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE).

[20]  Julie A. Shah,et al.  Decision-making authority, team efficiency and human worker satisfaction in mixed human–robot teams , 2015, Auton. Robots.

[21]  Stefan Leutenegger,et al.  SceneNet RGB-D: Can 5M Synthetic Images Beat Generic ImageNet Pre-training on Indoor Segmentation? , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[22]  Timothy S. McNerney.  From turtles to Tangible Programming Bricks: explorations in physical language design , 2004, Personal and Ubiquitous Computing.

[23]  Martin Buss,et al.  Human-Robot Collaboration: a Survey , 2008, Int. J. Humanoid Robotics.

[24]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[25]  Fabio Paternò,et al.  Personalization of Context-Dependent Applications Through Trigger-Action Rules , 2017, ACM Trans. Comput. Hum. Interact..

[26]  Alexandr Klimchik,et al.  Human-robot interaction for robotic manipulator programming in Mixed Reality , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[27]  Gregory D. Hager,et al.  CoSTAR: Instructing collaborative robots with behavior trees and vision , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[28]  Steven L. Tanimoto,et al.  A perspective on the evolution of live programming , 2013, 2013 1st International Workshop on Live Programming (LIVE).

[29]  Sorin Lerner,et al.  Small-Step Live Programming by Example , 2020, UIST.

[30]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[31]  Roman Weitschat,et al.  RAZER—A HRI for Visual Task-Level Programming and Intuitive Skill Parameterization , 2018, IEEE Robotics and Automation Letters.

[32]  Stefan Schaal,et al.  Robot Programming by Demonstration , 2009, Springer Handbook of Robotics.

[33]  JUSTINE ZHANG,et al.  Quantifying the Causal Effects of Conversational Tendencies , 2020, Proc. ACM Hum. Comput. Interact..

[34]  Allison Sauppé,et al.  Authoring and Verifying Human-Robot Interactions , 2018, UIST.

[35]  Michal Kapinus,et al.  Spatially Situated End-User Robot Programming in Augmented Reality , 2019, 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN).

[36]  Roland Siegwart,et al.  Visual Programming Language for Thymio II Robot , 2014 .

[37]  Maya Cakmak,et al.  Code3: A System for End-to-End Programming of Mobile Manipulator Robots for Novices and Experts , 2017, 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI.

[38]  Maya Cakmak,et al.  Supporting mental model accuracy in trigger-action programming , 2015, UbiComp.

[39]  Sonia Chernova,et al.  Leveraging depth data in remote robot teleoperation interfaces for general object manipulation , 2019, Int. J. Robotics Res..

[40]  Carlos Martínez,et al.  Easy robot programming concepts: An industrial perspective , 2013, 2013 IEEE International Conference on Automation Science and Engineering (CASE).

[41]  C GombolayMatthew,et al.  Decision-making authority, team efficiency and human worker satisfaction in mixed human---robot teams , 2015 .

[42]  Guy Hoffman,et al.  Evaluating Fluency in Human–Robot Collaboration , 2019, IEEE Transactions on Human-Machine Systems.

[43]  Sotiris Makris,et al.  Human–robot interaction review and challenges on task planning and programming , 2016, Int. J. Comput. Integr. Manuf..

[44]  Miguel Campusano,et al.  Live Robot Programming: The language, its implementation, and robot API independence , 2017, Sci. Comput. Program..

[45]  Clare Dixon,et al.  Formal Specification and Verification of Autonomous Robotic Systems , 2018, ACM Comput. Surv..

[46]  Blase Ur,et al.  How Users Interpret Bugs in Trigger-Action Programming , 2019, CHI.

[47]  Andreas Dömel,et al.  RAFCON: A graphical tool for engineering complex, robotic tasks , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).