BWIBots: A platform for bridging the gap between AI and human–robot interaction research

Recent progress in both AI and robotics have enabled the development of general purpose robot platforms that are capable of executing a wide variety of complex, temporally extended service tasks in open environments. This article introduces a novel, custom-designed multi-robot platform for research on AI, robotics, and especially human–robot interaction for service robots. Called BWIBots, the robots were designed as a part of the Building-Wide Intelligence (BWI) project at the University of Texas at Austin. The article begins with a description of, and justification for, the hardware and software design decisions underlying the BWIBots, with the aim of informing the design of such platforms in the future. It then proceeds to present an overview of various research contributions that have enabled the BWIBots to better (a) execute action sequences to complete user requests, (b) efficiently ask questions to resolve user requests, (c) understand human commands given in natural language, and (d) understand human intention from afar. The article concludes with a look forward towards future research opportunities and applications enabled by the BWIBot platform.

[1]  Danny De Schreye,et al.  Answer Set Planning , 1999 .

[2]  J.K. Aggarwal,et al.  Human activity analysis , 2011, ACM Comput. Surv..

[3]  Rodney D. Nielsen,et al.  Grounding the Meaning of Words through Vision and Interactive Gameplay , 2015, IJCAI.

[4]  Fangkai Yang,et al.  Mobile Robot Planning Using Action Language BC with an Abstraction Hierarchy , 2015, LPNMR.

[5]  Ross A. Knepper,et al.  Asking for Help Using Inverse Semantics , 2014, Robotics: Science and Systems.

[6]  Luca Iocchi,et al.  RoboCup@Home: Scientific Competition and Benchmarking for Domestic Service Robots , 2009 .

[7]  Xiaoping Chen,et al.  Extending C+ with Composite Actions for Robotic Task Planning , 2012, ICLP.

[8]  Jake K. Aggarwal,et al.  Robot-Centric Activity Recognition 'in the Wild' , 2015, ICSR.

[9]  Fangkai Yang,et al.  Planning in Action Language BC while Learning Action Costs for Mobile Robots , 2014, ICAPS.

[10]  Peter Stone,et al.  Multirobot Symbolic Planning under Temporal Uncertainty , 2017, AAMAS.

[11]  Jayant Krishnamurthy,et al.  Jointly Learning to Parse and Perceive: Connecting Natural Language to the Physical World , 2013, TACL.

[12]  Luc Steels,et al.  Co-Acquisition of Syntax and Semantics - An Investigation in Spatial Language , 2015, IJCAI.

[13]  T. Eiter,et al.  Answer Set Planning under Action Costs , 2002, JELIA.

[14]  Xiaoping Chen,et al.  Toward open knowledge enabling for human-robot interaction , 2013, HRI 2013.

[15]  Kai Oliver Arras,et al.  Inverse Reinforcement Learning algorithms and features for robot navigation in crowds: An experimental comparison , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[16]  Enrico Giunchiglia,et al.  Nonmonotonic causal theories , 2004, Artif. Intell..

[17]  Bernhard Nebel,et al.  In Defense of PDDL Axioms , 2003, IJCAI.

[18]  Nico Blodow,et al.  Fast Point Feature Histograms (FPFH) for 3D registration , 2009, 2009 IEEE International Conference on Robotics and Automation.

[19]  Wolfgang Faber.  Answer Set Programming , 2013, Reasoning Web.

[20]  Kai Oliver Arras,et al.  Multi-Hypothesis Social Grouping and Tracking for Mobile Robots , 2013, Robotics: Science and Systems.

[21]  Manuela M. Veloso,et al.  Dynamic User Task Scheduling for Mobile Robots , 2011, Automated Action Planning for Autonomous Mobile Robots.

[22]  Joohyung Lee,et al.  Cplus 2ASP: Computing Action Language ${\cal C}$ + in Answer Set Programming , 2013, LPNMR.

[23]  S Cousins,et al.  ROS on the PR2 [ROS Topics] , 2010 .

[24]  Wolfram Burgard,et al.  Monte Carlo Localization: Efficient Position Estimation for Mobile Robots , 1999, AAAI/IAAI.

[25]  J. Finger,et al.  Exploiting constraints in design synthesis , 1987 .

[26]  Peter Stone,et al.  A synthesis of automated planning and reinforcement learning for efficient, robust decision-making , 2016, Artif. Intell..

[27]  Matthew R. Walter,et al.  Approaching the Symbol Grounding Problem with Probabilistic Graphical Models , 2011, AI Mag..

[28]  Daniele Nardi,et al.  On-line semantic mapping , 2013, 2013 16th International Conference on Advanced Robotics (ICAR).

[29]  Peter Stone,et al.  Multi-Robot Human Guidance Using Topological Graphs , 2014, AAAI Spring Symposia.

[30]  Stephanie Rosenthal,et al.  CoBots: Robust Symbiotic Autonomous Mobile Service Robots , 2015, IJCAI.

[31]  Joohyung Lee,et al.  Cplus 2 ASP : Computing Action Language C + in Answer Set Programming , 2013 .

[32]  Luke S. Zettlemoyer,et al.  Learning from Unscripted Deictic Gesture and Language for Human-Robot Interactions , 2014, AAAI.

[33]  Michael Luck,et al.  Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1 - Volume 1 , 2010, AAMAS 2010.

[34]  Jayant Krishnamurthy,et al.  Toward Interactive Grounded Language Acqusition , 2013, Robotics: Science and Systems.

[35]  Esra Erdem,et al.  Answer set programming for collaborative housekeeping robotics: representation, reasoning, and execution , 2012, Intell. Serv. Robotics.

[36]  Peter Stone,et al.  Learning to Order Objects Using Haptic and Proprioceptive Exploratory Behaviors , 2016, IJCAI.

[37]  Scott Kuindersma,et al.  Dexterous mobility with the uBot-5 mobile manipulator , 2009, 2009 International Conference on Advanced Robotics.

[38]  J. Nelson Rushton,et al.  Probabilistic reasoning with answer sets , 2004, Theory and Practice of Logic Programming.

[39]  Peter Schüller,et al.  Finding optimal plans for multiple teams of robots through a mediator: A logic-based approach , 2013, Theory and Practice of Logic Programming.

[40]  Radu Bogdan Rusu,et al.  3D is here: Point Cloud Library (PCL) , 2011, 2011 IEEE International Conference on Robotics and Automation.

[41]  Joohyung Lee,et al.  Action Language BC+: Preliminary Report , 2015, AAAI.

[42]  Oussama Khatib,et al.  Elastic bands: connecting path planning and control , 1993, [1993] Proceedings IEEE International Conference on Robotics and Automation.

[43]  Demetri Terzopoulos,et al.  Snakes: Active contour models , 2004, International Journal of Computer Vision.

[44]  References , 1971 .

[45]  Nick Hawes,et al.  Task scheduling for mobile robots using interval algebra , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[46]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[47]  Andrew Howard,et al.  Design and use paradigms for Gazebo, an open-source multi-robot simulator , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[48]  Peter Stone,et al.  Learning Multi-Modal Grounded Linguistic Semantics by Playing "I Spy" , 2016, IJCAI.

[49]  Zicheng Liu,et al.  HON4D: Histogram of Oriented 4D Normals for Activity Recognition from Depth Sequences , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  John McCarthy,et al.  SOME PHILOSOPHICAL PROBLEMS FROM THE STANDPOINT OF ARTI CIAL INTELLIGENCE , 1987 .

[51]  Matteo Munaro,et al.  Fast RGB-D people tracking for service robots , 2014, Auton. Robots.

[52]  Connor Schenck,et al.  Grounding semantic categories in behavioral interactions: Experiments with 100 objects , 2014, Robotics Auton. Syst..

[53]  Larry H. Matthies,et al.  First-Person Activity Recognition: What Are They Doing to Me? , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[54]  Marc Hanheide,et al.  Robot task planning and explanation in open and uncertain worlds , 2017, Artif. Intell..

[55]  Xiaoping Chen,et al.  Developing high-level cognitive functions for service robots , 2010, AAMAS.

[56]  Michael Gelfond,et al.  Classical negation in logic programs and disjunctive databases , 1991, New Generation Computing.

[57]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[58]  Jake K. Aggarwal,et al.  Robot-centric Activity Recognition from First-Person RGB-D Videos , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[59]  Chandler Jake,et al.  Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) , 2016 .

[60]  Esra Erdem,et al.  Applications of Action Languages in Cognitive Robotics , 2012, Correct Reasoning.

[61]  Peter Stone,et al.  Leading the Way: An Efficient Multi-robot Guidance System , 2015, AAMAS.

[62]  D. You.  Asking for help. , 2003, Nursing standard (Royal College of Nursing (Great Britain) : 1987).

[63]  Ross A. Knepper,et al.  Herb 2.0: Lessons Learned From Developing a Mobile Manipulator for the Home , 2012, Proceedings of the IEEE.

[64]  Bernhard Nebel,et al.  The FF Planning System: Fast Plan Generation Through Heuristic Search , 2011, J. Artif. Intell. Res..

[65]  Milica Gasic,et al.  POMDP-Based Statistical Spoken Dialog Systems: A Review , 2013, Proceedings of the IEEE.

[66]  Jake K. Aggarwal,et al.  Robot-Centric Activity Prediction from First-Person Videos: What Will They Do to Me? , 2015, 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[67]  James F. Allen,et al.  SALL-E: Situated Agent for Language Learning , 2013, AAAI.

[68]  Paul Lamere,et al.  Sphinx-4: a flexible open source framework for speech recognition , 2004 .

[69]  Luke S. Zettlemoyer,et al.  Bootstrapping Semantic Parsers from Conversations , 2011, EMNLP.

[70]  Jake K. Aggarwal,et al.  View invariant human action recognition using histograms of 3D joints , 2012, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[71]  Luke S. Zettlemoyer,et al.  A Joint Model of Language and Perception for Grounded Attribute Learning , 2012, ICML.

[72]  Wolfram Burgard,et al.  Improved Techniques for Grid Mapping With Rao-Blackwellized Particle Filters , 2007, IEEE Transactions on Robotics.

[73]  Esra Erdem,et al.  Bridging the Gap between High-Level Reasoning and Low-Level Control , 2009, LPNMR.

[74]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[75]  Stephanie Rosenthal,et al.  An effective personal mobile robot agent through symbiotic human-robot interaction , 2010, AAMAS.

[76]  Kai Zhan,et al.  First-Person Activity Recognition , 2014 .

[77]  Martin Gebser,et al.  Coala: A Compiler from Action Languages to ASP , 2010, JELIA.

[78]  Peter Stone,et al.  CORPP: Commonsense Reasoning and Probabilistic Planning, as Applied to Dialog with a Mobile Robot , 2015, AAAI.

[79]  Ilkka Niemelä,et al.  Logic programs with stable model semantics as a constraint programming paradigm , 1999, Annals of Mathematics and Artificial Intelligence.

[80]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[81]  Kai Chen,et al.  The Intelligent Techniques in Robot KeJia - The Champion of RoboCup@Home 2014 , 2014, RoboCup.

[82]  Malte Helmert,et al.  The Fast Downward Planning System , 2006, J. Artif. Intell. Res..

[83]  Paul Taylor,et al.  The architecture of the Festival speech synthesis system , 1998, SSW.

[84]  Luke S. Zettlemoyer,et al.  Learning to Parse Natural Language Commands to a Robot Control System , 2012, ISER.

[85]  Ángel García Olaya,et al.  Control of autonomous mobile robots with automated planning , 2011 .

[86]  Alexander Verl,et al.  Care-O-bot® 3 - creating a product vision for service robot applications by integrating design and technology , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[87]  Kurt Konolige,et al.  The Office Marathon: Robust navigation in an indoor office environment , 2010, 2010 IEEE International Conference on Robotics and Automation.

[88]  T. Power.  Play and Exploration in Children and Animals , 1999 .

[89]  Esra Erdem,et al.  A case study on the Tower of Hanoi challenge: Representation, reasoning and execution , 2013, 2013 IEEE International Conference on Robotics and Automation.

[90]  Cassandra Mariette Carley.  Human Activity Analysis , 2018 .

[91]  Marius Thomas Lindauer,et al.  Potassco: The Potsdam Answer Set Solving Collection , 2011, AI Commun..

[92]  Manuela M. Veloso,et al.  Learning environmental knowledge from task-based human-robot dialog , 2013, 2013 IEEE International Conference on Robotics and Automation.

[93]  Peter Stone,et al.  Learning to Interpret Natural Language Commands through Human-Robot Dialog , 2015, IJCAI.

[94]  Marwan Torki,et al.  Human Action Recognition Using a Temporal Hierarchy of Covariance Descriptors on 3D Joint Locations , 2013, IJCAI.

[95]  Michael Gelfond,et al.  Knowledge Representation, Reasoning, and the Design of Intelligent Agents: The Answer-Set Programming Approach , 2014 .

[96]  Victor W. Marek,et al.  Stable models and an alternative logic programming paradigm , 1998, The Logic Programming Paradigm.

[97]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[98]  Balaraman Ravindran,et al.  Activity Recognition for Natural Human Robot Interaction , 2014, ICSR.

[99]  Mohan Sridharan,et al.  Mixed Logical Inference and Probabilistic Planning for Robots in Unreliable Worlds , 2015, IEEE Transactions on Robotics.