A multimodal interface for real-time soldier-robot teaming

Recent research and advances in robotics have led to the development of novel platforms leveraging new sensing capabilities for semantic navigation. As these systems becoming increasingly more robust, they support highly complex commands beyond direct teleoperation and waypoint finding facilitating a transition away from robots as tools to robots as teammates. Supporting future Soldier-Robot teaming requires communication capabilities on par with human-human teams for successful integration of robots. Therefore, as robots increase in functionality, it is equally important that the interface between the Soldier and robot advances as well. Multimodal communication (MMC) enables human-robot teaming through redundancy and levels of communications more robust than single mode interaction. Commercial-off-the-shelf (COTS) technologies released in recent years for smart-phones and gaming provide tools for the creation of portable interfaces incorporating MMC through the use of speech, gestures, and visual displays. However, for multimodal interfaces to be successfully used in the military domain, they must be able to classify speech, gestures, and process natural language in real-time with high accuracy. For the present study, a prototype multimodal interface supporting real-time interactions with an autonomous robot was developed. This device integrated COTS Automated Speech Recognition (ASR), a custom gesture recognition glove, and natural language understanding on a tablet. This paper presents performance results (e.g. response times, accuracy) of the integrated device when commanding an autonomous robot to perform reconnaissance and surveillance activities in an unknown outdoor environment.

[1]  Jean Oh,et al.  An Integrated Assessment of Progress in Robotic Perception and Semantic Navigation , 2015 .

[2]  Randall Shumaker,et al.  Virtual, Augmented and Mixed Reality. Designing and Developing Virtual and Augmented Environments , 2014, Lecture Notes in Computer Science.

[3]  Jonathan Harris,et al.  Technological evaluation of gesture and speech interfaces for enabling dismounted soldier-robot dialogue , 2016, SPIE Defense + Security.

[4]  Volker Graefe,et al.  Dependable multimodal communication and interaction with robotic assistants , 2002, Proceedings. 11th IEEE International Workshop on Robot and Human Interactive Communication.

[5]  Florian Jentsch,et al.  From Tools to Teammates , 2011 .

[6]  Florian Jentsch,et al.  Field Assessment of Multimodal Communication for Dismounted Human-Robot Teams , 2015 .

[7]  Joseph Mariani.  Spoken language processing , 2009 .

[8]  Stefanie Tellex,et al.  A natural language planner interface for mobile manipulators , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[9]  Lisa A. Parr,et al.  Perceptual biases for multimodal cues in chimpanzee (Pan troglodytes) affect recognition , 2004, Animal Cognition.

[10]  Matthew R. Walter,et al.  Understanding Natural Language Commands for Robotic Navigation and Mobile Manipulation , 2011, AAAI.

[11]  Florian Jentsch,et al.  Development of a Squad Level Vocabulary for Human-Robot Interaction , 2014, HCI.

[12]  Robert Dean.  Common world model for unmanned systems , 2013, Defense, Security, and Sensing.

[13]  P. Marler,et al.  Communication Goes Multimodal , 1999, Science.

[14]  Lauren Reinerman-Jones,et al.  Visual and tactile interfaces for bi-directional human robot communication , 2013, Defense, Security, and Sensing.

[15]  Jean Oh,et al.  Common world model for unmanned systems: Phase 2 , 2014, Defense + Security Symposium.

[16]  Jonathan Harris,et al.  Speech and gesture interfaces for squad-level human-robot teaming , 2014, Defense + Security Symposium.

[17]  Matthew R. Walter,et al.  On the performance of hierarchical distributed correspondence graphs for efficient symbol grounding of robot instructions , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[18]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[19]  Matthew R. Walter,et al.  Learning models for following natural language directions in unknown environments , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[20]  Charles A. DiBerardino,et al.  Robotic collaborative technology alliance: an open architecture approach to integrated research , 2014, Defense + Security Symposium.