Interactive Multi-Modal Robot Programming

This paper introduces a novel approach to program a robot interactively through a multi-modal interface. The key characteristic of this approach is that the user can provide feedback interactively at any time - during both the programming and the execution phase. The framework takes a three-step approach to the problem: multi-modal recognition, intention interpretation, and prioritized task execution. The multi-modal recognition module translates hand gestures and spontaneous speech into a structured symbolic data stream without abstracting away the user's intent. The intention interpretation module selects the appropriate primitives to generate a task based on the user's input, the system's current state, and robot sensor data. Finally, the prioritized task execution module selects and executes skill primitives based on the system's current state, sensor inputs, and prior tasks. The framework is demonstrated by interactively controlling and programming a vacuum-cleaning robot.

[1]  Manuela Veloso,et al.  Automated Robot Behavior Recognition Applied to Robotic Soccer , 1999 .

[2]  Giancarlo Iannizzotto,et al.  VirtualBoard: real-time visual gesture recognition for natural human-computer interaction , 2003, Proceedings International Parallel and Distributed Processing Symposium.

[3]  Alex Pentland,et al.  Real-time American Sign Language recognition from video using hidden Markov models , 1995 .

[4]  Alexander Zelinsky,et al.  An algorithm for real-time stereo vision implementation of head pose and gaze direction measurement , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[5]  Pradeep K. Khosla,et al.  A human machine interface for distributed virtual laboratories , 1994, IEEE Robotics & Automation Magazine.

[6]  Horst-Michael Groß,et al.  Neural Architecture for Gesture-Based Human-Machine-Interaction , 1997, Gesture Workshop.

[7]  T. Ishida,et al.  A small biped entertainment robot SDR-4X II , 2003, Proceedings 2003 IEEE International Symposium on Computational Intelligence in Robotics and Automation. Computational Intelligence in Robotics and Automation for the New Millennium (Cat. No.03EX694).

[8]  Illah Nourbakhsh,et al.  Path planning for the Cye personal robot , 2000, Proceedings. 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2000) (Cat. No.00CH37113).

[9]  Pradeep K. Khosla,et al.  Tropism-based cognition for the interpretation of context-dependent gestures , 1997, Proceedings of International Conference on Robotics and Automation.

[10]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[11]  Francisco Javier Caminero Gil,et al.  On-line garbage modeling with discriminant analysis for utterance verification , 1996, Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP '96.

[12]  Richard M. Voyles,et al.  Gesture-based programming for robotics: human-augmented software adaptation , 1999, IEEE Intell. Syst..

[13]  Pradeep K. Khosla,et al.  A software architecture-based human-machine interface for reconfigurable sensor-based control systems , 1993, Proceedings of 8th IEEE International Symposium on Intelligent Control.

[14]  Sharon L. Oviatt,et al.  Taming recognition errors with a multimodal interface , 2000, CACM.

[15]  Magdalena D. Bugajska,et al.  Building a Multimodal Human-Robot Interface , 2001, IEEE Intell. Syst..

[16]  Rüdiger Dillmann,et al.  Interactive Robot Programming Based on Human Demonstration and Advice , 1998, Sensor Based Intelligent Robots.

[17]  Pradeep K. Khosla,et al.  Inferring user intent for learning by observation , 2004 .

[18]  Steve Young,et al.  Token passing: a simple conceptual model for connected speech recognition systems , 1989 .

[19]  Francis K. H. Quek,et al.  Toward a vision-based hand gesture interface , 1994 .

[20]  Michael P. Deisenroth,et al.  An interactive programming system for the IBM 7545 robot , 1987 .

[21]  H. Kimura,et al.  Acquiring hand-action models in task and behavior levels by a learning robot through observing human demonstrations , 2000 .

[22]  H.,et al.  Token Passing : a Simple Conceptual Model for ConnectedSpeech Recognition , 1989 .

[23]  R. Dillmann,et al.  Programming service tasks in household environments by human demonstration , 2002, Proceedings. 11th IEEE International Workshop on Robot and Human Interactive Communication.

[24]  Alan C. Schultz,et al.  Goal tracking in a natural language interface: towards achieving adjustable autonomy , 1999, Proceedings 1999 IEEE International Symposium on Computational Intelligence in Robotics and Automation. CIRA'99 (Cat. No.99EX375).

[25]  Terrence Fong,et al.  Novel interfaces for remote driving: gesture, haptic, and PDA , 2001, SPIE Optics East.

[26]  Narendra Ahuja,et al.  Face Detection and Gesture Recognition for Human-Computer Interaction , 2001, The International Series in Video Computing.

[27]  Guido Bugmann,et al.  Training Personal Robots Using Natural Language Instruction , 2001, IEEE Intell. Syst..

[28]  Yoji Yamada,et al.  Human error recovery for a human/robot parts conveyance system , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[29]  Marjorie Skubic,et al.  Communicating with Teams of Cooperative Robots , 2002 .

[30]  Takuichi Nishimura,et al.  Adaptation to gesture performers by an on-line teaching system for spotting recognition of gestures from a time-varying image , 2000, Systems and Computers in Japan.

[31]  Michael Vande Weghe,et al.  An architecture for gesture-based control of mobile robots , 1999, Proceedings 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human and Environment Friendly Robots with High Intelligence and Emotional Quotients (Cat. No.99CH36289).

[32]  Alex Pentland,et al.  Understanding purposeful human motion , 1999, Proceedings IEEE International Workshop on Modelling People. MPeople'99.

[33]  R. Dillmann,et al.  Using gesture and speech control for commanding a robot assistant , 2002, Proceedings. 11th IEEE International Workshop on Robot and Human Interactive Communication.

[34]  M. Schofield,et al.  "Neither master nor slave...". A practical case study in the development and employment of cleaning robots , 1999, 1999 7th IEEE International Conference on Emerging Technologies and Factory Automation. Proceedings ETFA '99 (Cat. No.99TH8467).

[35]  Toshi Takamori,et al.  Multi-modal human robot interaction for map generation , 2001, Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the the Next Millennium (Cat. No.01CH37180).

[36]  Pradeep K. Khosla,et al.  Manipulation task primitives for composing robot skills , 1997, Proceedings of International Conference on Robotics and Automation.

[37]  Thomas B. Sheridan,et al.  Plan recognition and generalization in command languages with application to telerobotics , 1991, IEEE Trans. Syst. Man Cybern..

[38]  Thea Iberall,et al.  Dextrous robot hands , 1990 .

[39]  Charles E. Thorpe,et al.  Collaborative control: a robot-centric model for vehicle teleoperation , 2001 .

[40]  Masahiro Fujita,et al.  Motion entertainment by a small humanoid robot based on OPEN-R , 2001, Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the the Next Millennium (Cat. No.01CH37180).

[41]  P. K. Khosla,et al.  Iconic language for reconfigurable sensor-based control systems , 1994 .

[42]  Narendra Ahuja,et al.  Extraction of 2D Motion Trajectories and Its Application to Hand Gesture Recognition , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  Kazuhiko Kawamura,et al.  Towards a Unified Framework for Human-Humanoid Interaction , 2000 .

[44]  Yangsheng Xu,et al.  Online, interactive learning of gestures for human/robot interfaces , 1996, Proceedings of IEEE International Conference on Robotics and Automation.

[45]  Kosei Kitagaki,et al.  A Telerobotics System using Planning Functions Based on Manipulation Skills and Teaching-by-Demonastration Technique in VR , 2000 .

[46]  Christiaan J. J. Paredis,et al.  Interactive multi-modal robot programming , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[47]  R. Stallman.  EMACS the extensible, customizable self-documenting display editor , 1981, SIGPLAN SIGOA Symposium on Text Manipulation.

[48]  Mei-Yuh Hwang,et al.  The SPHINX-II speech recognition system: an overview , 1993, Comput. Speech Lang..

[49]  Jean Ponce,et al.  Computer Vision: A Modern Approach , 2002 .

[50]  Vladimir Pavlovic,et al.  Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[51]  Abhijit Nagchaudhuri,et al.  LEGO robotics products boost student creativity in precollege programs at UMES , 2002, 32nd Annual Frontiers in Education.

[52]  John J. Craig,et al.  Industrial robot programming languages: A comparative evaluation , 1984, IEEE Transactions on Systems, Man, and Cybernetics.

[53]  Yoshiaki Shirai,et al.  Interactive gesture interface for intelligent wheelchairs , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[54]  Christiaan J. J. Paredis,et al.  Intention aware interactive multi-modal robot programming , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[55]  Y. Matsumotot,et al.  Development of intelligent wheelchair system with face and gaze based interface , 2001, Proceedings 10th IEEE International Workshop on Robot and Human Interactive Communication. ROMAN 2001 (Cat. No.01TH8591).

[56]  Paul E. Rybski,et al.  Interactive task training of a mobile robot through human gesture recognition , 1999, Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C).

[57]  Yoji Yamada,et al.  Construction of a human/robot coexistence system based on a model of human will-intention and desire , 1999, Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C).

[58]  D. Kortenkamp,et al.  Distributed, autonomous control of Space habitats , 2001, 2001 IEEE Aerospace Conference Proceedings (Cat. No.01TH8542).

[59]  Pradeep K. Khosla,et al.  Gesture-based programming: a preliminary demonstration , 1999, Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C).

[60]  Katsushi Ikeuchi,et al.  Task-model based human robot cooperation using vision , 1999, Proceedings 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human and Environment Friendly Robots with High Intelligence and Emotional Quotients (Cat. No.99CH36289).

[61]  Masahiro Fujita,et al.  AIBO: Toward the Era of Digital Creatures , 2001, Int. J. Robotics Res..

[62]  Vijay Kumar,et al.  Human robot interaction: application to smart wheelchairs , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[63]  Claus Bahlmann,et al.  Measuring HMM similarity with the Bayes probability of error and its application to online handwriting recognition , 2001, Proceedings of Sixth International Conference on Document Analysis and Recognition.

[64]  Herman Roodenburg,et al.  A Cultural History of Gesture , 1992 .

[65]  Paul A. Beardsley,et al.  Computer Vision for Interactive Computer Graphics , 1998, IEEE Computer Graphics and Applications.

[66]  K. Mardia.  Statistics of Directional Data , 1972 .

[67]  Katsushi Ikeuchi,et al.  Toward automatic robot instruction from perception-mapping human grasps to manipulator grasps , 1997, IEEE Trans. Robotics Autom..

[68]  Lynne E. Parker,et al.  Multi-Robot Systems: From Swarms to Intelligent Automata , 2002, Springer Netherlands.

[69]  ThrunSebastian,et al.  A Gesture Based Interface for Human-Robot Interaction , 2000 .

[70]  Marjorie Skubic,et al.  Using spatial language in a human-robot dialog , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[71]  Lynn Conway,et al.  Video mirroring and iconic gestures: enhancing basic videophones to provide visual coaching and visual control , 1998 .

[72]  H. Harry Asada,et al.  Hand-in-glove human-machine interface and interactive control: task process modeling using dual Petri nets , 1998, Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No.98CH36146).

[73]  Kazuo Tanie,et al.  Human-machine interaction through an intelligent user interface based on contention architecture , 1996, Proceedings 5th IEEE International Workshop on Robot and Human Communication. RO-MAN'96 TSUKUBA.

[74]  Katsushi Ikeuchi,et al.  Toward an assembly plan from observation. I. Task recognition with polyhedral objects , 1994, IEEE Trans. Robotics Autom..

[75]  George Musser.  Robots That Suck , 2003 .

[76]  Takuichi Nishimura,et al.  Adaptation to gesture performers by an on‐line teaching system for spotting recognition of gestures from a time‐varying image , 2000 .

[77]  Rüdiger Dillmann,et al.  Learning Robot Behaviour and Skills Based on Human Demonstration and Advice: The Machine Learning Paradigm , 2000 .

[78]  Hiroaki Kitano,et al.  Development of an Autonomous Quadruped Robot for Robot Entertainment , 1998, Auton. Robots.

[79]  Robert J. Anderson,et al.  A telerobot control system for accident response , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[80]  David Kortenkamp,et al.  Recognizing and Interpreting Gestures on a Mobile Robot , 1996, AAAI/IAAI, Vol. 2.

[81]  Wolfram Burgard,et al.  MINERVA: a second-generation museum tour-guide robot , 1999, Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C).