High-Level Motor Planning Assessment During Performance of Complex Action Sequences in Humans and a Humanoid Robot

Examining complex cognitive-motor performance in humanoid robots and humans can inform their interactions in a social context of team dynamics. Namely, the understanding of human cognitive-motor control and learning mechanisms can inform human motor behavior and also the development of intelligent controllers for robots when interacting with people. While prior humans and humanoid robot studies mainly examined motion planning, only a few have investigated high-level motor planning underlying action sequences for complex task execution. This sparse work has largely considered well-constrained problems using fairly simple performance assessment methods without detailed action sequence analyses. Here we qualitatively and quantitatively assess action sequences generated by humans and a humanoid robot during execution of two tasks providing various challenge levels and learning paradigms while offering flexible success criteria. The Levenshtein distance and its operators are adapted to the motor domain to provide a detailed performance assessment of action sequences by comparing them to a reference sequence (perfect sequence having a minimal number of actions). The results reveal that (i) humans produced a large variety of action sequences combining perfect and imperfect sequences while still reaching the task goal, whereas the robot generated perfect/near-perfect successful action sequences; (ii) the Levenshtein distance and the number of insertions provide reliable performance markers capable of differentiating perfect and imperfect sequences; (iii) the deletion operator is the most sensitive marker of action sequence failure. This work complements prior efforts for complex task performance assessment in humans and humanoid robots and has the potential to inform human–machine interactions.

[1]  Shlomo Zilberstein,et al.  Integration of Planning with Recognition for Responsive Interaction Using Classical Planners , 2017, AAAI.

[2]  Bi Liu,et al.  A Normalized Levenshtein Distance Metric , 2007, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Rodolphe J. Gentili,et al.  Changes in motor performance and mental workload during practice of reaching movements: a team dynamics perspective , 2017, Experimental Brain Research.

[4]  E. Vakil,et al.  Baseline performance and learning rate of conceptual and perceptual skill-learning tasks: The effect of moderate to severe traumatic brain injury , 2014, Journal of clinical and experimental neuropsychology.

[5]  Matthew W. Miller,et al.  Combined assessment of attentional reserve and cognitive-motor effort under various levels of challenge with a dry EEG system. , 2018, Psychophysiology.

[6]  James A. Reggia,et al.  An Object-Centric Paradigm for Robot Programming by Demonstration , 2015, HCI.

[7]  Moritz Tenorth,et al.  Representation and Exchange of Knowledge About Actions, Objects, and Environments in the RoboEarth Framework , 2013, IEEE Transactions on Automation Science and Engineering.

[8]  Jason M. Chein,et al.  Primacy and recency effects as indices of the focus of attention , 2014, Front. Hum. Neurosci..

[9]  Michael J. Fischer,et al.  The String-to-String Correction Problem , 1974, JACM.

[10]  R. Brent Gillespie,et al.  Architectures for Shared Control of Vehicle Steering , 2016 .

[11]  N. Cowan.  The magical number 4 in short-term memory: A reconsideration of mental storage capacity , 2001, Behavioral and Brain Sciences.

[12]  Rodolphe J. Gentili,et al.  The effects of team environment on cerebral cortical processes and attentional reserve. , 2014 .

[13]  R. Magill.  Motor learning and control : concepts and applications , 2004 .

[14]  Cynthia Breazeal,et al.  Improved human-robot team performance using Chaski, A human-inspired plan execution system , 2011, 2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[15]  J. Saint-Cyr,et al.  Procedural learning and neostriatal dysfunction in man. , 1988, Brain : a journal of neurology.

[16]  Siddhartha S. Srinivasa,et al.  Human-robot mutual adaptation in collaborative tasks: Models and experiments , 2017, Int. J. Robotics Res..

[17]  Edoardo Battaglia,et al.  A Review of Intent Detection, Arbitration, and Communication Aspects of Shared Control for Physical Human-Robot Interaction , 2018 .

[18]  Moritz Tenorth,et al.  Automated alignment of specifications of everyday manipulation tasks , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[19]  I. René J. A. te Boekhorst,et al.  Learning about natural human-robot interaction styles , 2006, Robotics Auton. Syst..

[20]  J. Mathieu,et al.  The influence of shared mental models on team process and performance. , 2000, The Journal of applied psychology.

[21]  Vladimir Popov,et al.  On the adjustment of the weights of the Levenshtein distance for the description of sequences of rhythmic motor primitives , 2015 .

[22]  Stefanos Nikolaidis,et al.  Efficient Model Learning from Joint-Action Demonstrations for Human-Robot Collaborative Tasks , 2015, 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[23]  Toyoaki Nishida,et al.  Controlling gaze with an embodied interactive control architecture , 2010, Applied Intelligence.

[24]  Theresa C. Hauge,et al.  A Novel Parsimonious Cause-Effect Reasoning Algorithm for Robot Imitation and Plan Recognition , 2018, IEEE Transactions on Cognitive and Developmental Systems.

[25]  Murdock,et al.  The serial position effect of free recall , 1962 .

[26]  Cynthia Breazeal,et al.  Learning From and About Others: Towards Using Imitation to Bootstrap the Social Understanding of Others by Robots , 2005, Artificial Life.

[27]  James A. Reggia,et al.  Imitation Learning as Cause-Effect Reasoning , 2016, AGI.

[28]  Matthew W. Miller,et al.  Measurement of attentional reserve and mental effort for cognitive workload assessment under various task demands during dual-task walking , 2018, Biological Psychology.

[29]  Brian Charles Williams,et al.  Concurrent Plan Recognition and Execution for Human-Robot Teams , 2014, ICAPS.

[30]  Li Yujian,et al.  A Normalized Levenshtein Distance Metric , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Rodolphe J. Gentili,et al.  Mental workload and motor performance dynamics during practice of reaching movements under various levels of task difficulty , 2017, Neuroscience.

[32]  Matthias Scheutz,et al.  Computational Mechanisms for Mental Models in Human-Robot Interaction , 2013, HCI.

[33]  Tal Oron-Gilad,et al.  Understanding and Resolving Failures in Human-Robot Interaction: Literature Review and Model Development , 2018, Front. Psychol..

[34]  Sahin Albayrak,et al.  Joint Human-Agent Activities: Challenges and Definition , 2016, MATES.

[35]  ThienLuan Ho,et al.  A parallel approximate string matching under Levenshtein distance on graphics processing units using warp-shuffle operations , 2017, PloS one.

[36]  E. Todorov.  Optimality principles in sensorimotor control , 2004, Nature Neuroscience.

[37]  Katherine J. Kuchenbecker,et al.  Qualitative User Reactions to a Hand-Clapping Humanoid Robot , 2016, ICSR.

[38]  J. Grafman,et al.  Are the frontal lobes implicated in “planning” functions? Interpreting data from the Tower of Hanoi , 1995, Neuropsychologia.

[39]  Mariëtte Huizinga,et al.  Tower of Hanoi disk-transfer task: Influences of strategy knowledge and learning on performance , 2005 .

[40]  Jean Scholtz,et al.  Common metrics for human-robot interaction , 2006, HRI '06.

[41]  G. Hirzinger,et al.  Self-guided robotic camera control for laparoscopic surgery compared with human camera control. , 1999, American journal of surgery.

[42]  Deborah Richards,et al.  The Impact of Multimodal Communication on a Shared Mental Model, Trust, and Commitment in Human-Intelligent Virtual Agent Teams , 2018, Multimodal Technol. Interact..

[43]  Leila Takayama,et al.  Communication and knowledge sharing in human-robot interaction and learning from demonstration , 2010, Neural Networks.

[44]  Thomas B. Sheridan,et al.  Human–Robot Interaction , 2016, Hum. Factors.

[45]  Guy Madison,et al.  Executive control and working memory are involved in sub-second repetitive motor timing , 2016, Experimental Brain Research.

[46]  J. Deese,et al.  Serial effects in recall of unorganized and sequentially organized verbal material. , 1957, Journal of experimental psychology.

[47]  Liz Sonenberg,et al.  Implicit Coordination Strategies for Effective Team Communication , 2016, Hum. Factors.

[48]  Yu Zhang,et al.  A human factors analysis of proactive support in human-robot teaming , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[49]  Y. Benjamini,et al.  Controlling the false discovery rate: a practical and powerful approach to multiple testing , 1995 .

[50]  James A. Reggia,et al.  A virtual demonstrator environment for robot imitation learning , 2015, 2015 IEEE International Conference on Technologies for Practical Robot Applications (TePRA).

[51]  Stefanos Nikolaidis,et al.  Efficient Model Learning for Human-Robot Collaborative Tasks , 2014, ArXiv.

[52]  Michael A. Peshkin,et al.  A general framework for cobot control , 2001, IEEE Trans. Robotics Autom..

[53]  Vladimir I. Levenshtein,et al.  Binary codes capable of correcting deletions, insertions, and reversals , 1965 .

[54]  Cynthia Breazeal,et al.  An Empirical Analysis of Team Coordination Behaviors and Action Planning With Application to Human–Robot Teaming , 2010, Hum. Factors.