Human modeling for human–robot collaboration

Teamwork is best achieved when members of the team understand one another. Human–robot collaboration poses a particular challenge to this goal due to the differences between individual team members, both mentally/computationally and physically. One way in which this challenge can be addressed is by developing explicit models of human teammates. Here, we discuss, compare and contrast the many techniques available for modeling human cognition and behavior, and evaluate their benefits and drawbacks in the context of human–robot collaboration.

[1]  Sangeet Khemlani,et al.  The processes of inference , 2013, Argument Comput..

[2]  L. Baum,et al.  An inequality and associated maximization technique in statistical estimation of probabilistic functions of a Markov process , 1972 .

[3]  Kee-Eung Kim,et al.  Inverse Reinforcement Learning in Partially Observable Environments , 2009, IJCAI.

[4]  Shuangquan Wang,et al.  Human activity recognition with user-free accelerometers in the sensor networks , 2005, 2005 International Conference on Neural Networks and Brain.

[5]  Danica Kragic,et al.  Adaptive Virtual Fixtures for Machine-Assisted Teleoperation Tasks , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[6]  Sangeet Khemlani,et al.  Immediate inferences from quantified assertions , 2015, Quarterly journal of experimental psychology.

[7]  C. Lebiere,et al.  An integrated theory of list memory. , 1998 .

[8]  Joshua B. Tenenbaum,et al.  Bayesian Theory of Mind: Modeling Joint Belief-Desire Attribution , 2011, CogSci.

[9]  Christopher W. Geib,et al.  Considering State in Plan Recognition with Lexicalized Grammars , 2012, CogRob@AAAI.

[10]  Ramakant Nevatia,et al.  Coupled Hidden Semi Markov Models for Activity Recognition , 2007, 2007 IEEE Workshop on Motion and Video Computing (WMVC'07).

[11]  Lynne E. Parker,et al.  4-dimensional local spatio-temporal features for human activity recognition , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[12]  Joshua B. Tenenbaum,et al.  Bayesian models of human action understanding , 2005, NIPS.

[13]  Candace L. Sidner,et al.  Using plan recognition in human-computer collaboration , 1999 .

[14]  Redwan Alqasemi,et al.  Telemanipulation Assistance Based on Motion Intention Recognition , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[15]  Aaron F. Bobick,et al.  Recognition of Visual Activities and Interactions by Stochastic Parsing , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[16]  P. Johnson-Laird,et al.  Children's creation of algorithms: simulations and gestures , 2016 .

[17]  Cynthia Breazeal,et al.  Social interactions in HRI: the robot view , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[18]  Brian Scassellati,et al.  Socially assistive robotics [Grand Challenges of Robotics] , 2007, IEEE Robotics & Automation Magazine.

[19]  Colin de la Higuera,et al.  Current Trends in Grammatical Inference , 2000, SSPR/SPR.

[20]  Fei-Fei Li,et al.  Large-Scale Video Classification with Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  Christian Laugier,et al.  Intentional motion on-line learning and prediction , 2008, Machine Vision and Applications.

[22]  Cynthia Breazeal,et al.  An Embodied Cognition Approach to Mindreading Skills for Socially Intelligent Robots , 2009, Int. J. Robotics Res..

[23]  Monique Thonnat,et al.  Activity Recognition from Video Sequences using Declarative Models , 2000, ECAI.

[24]  Irfan A. Essa,et al.  Recognizing multitasked activities from video using stochastic context-free grammar , 2002, AAAI/IAAI.

[25]  Svetha Venkatesh,et al.  Policy Recognition in the Abstract Hidden Markov Model , 2002, J. Artif. Intell. Res..

[26]  Stefanos Nikolaidis,et al.  Efficient Model Learning for Human-Robot Collaborative Tasks , 2014, ArXiv.

[27]  Nando de Freitas,et al.  Rao-Blackwellised Particle Filtering for Dynamic Bayesian Networks , 2000, UAI.

[28]  Wolfram Burgard,et al.  Learning Motion Patterns of People for Compliant Robot Motion , 2005, Int. J. Robotics Res..

[29]  Maja J. Mataric,et al.  Investigating Implicit Cues for User State Estimation in Human-Robot Interaction Using Physiological Measurements , 2007, RO-MAN 2007 - The 16th IEEE International Symposium on Robot and Human Interactive Communication.

[30]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[31]  P. Johnson-Laird How We Reason , 2006 .

[32]  Nate Blaylock,et al.  Hierarchical Goal Recognition , 2014 .

[33]  Sangeet Khemlani,et al.  Automating Human Inference , 2016, Bridging@IJCAI.

[34]  Michael Vande Weghe,et al.  An architecture for gesture-based control of mobile robots , 1999, Proceedings 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human and Environment Friendly Robots with High Intelligence and Emotional Quotients (Cat. No.99CH36289).

[35]  Sangeet Khemlani,et al.  How people differ in syllogistic reasoning , 2016, CogSci.

[36]  Eduardo Salas,et al.  Planning, Shared Mental Models, and Coordinated Performance: An Empirical Link Is Established , 1999, Hum. Factors.

[37]  Juan-Luis Gorricho,et al.  Activity Recognition from Accelerometer Data on a Mobile Phone , 2009, IWANN.

[38]  Changchun Liu,et al.  Online Affect Detection and Robot Behavior Adaptation for Intervention of Children With Autism , 2008, IEEE Transactions on Robotics.

[39]  Darryl W. Schneider,et al.  A memory-based model of Hick’s law , 2011, Cognitive Psychology.

[40]  Laura M. Hiatt,et al.  A Cognitive Model of Theory of Mind , 2010 .

[41]  Christian Szegedy,et al.  DeepPose: Human Pose Estimation via Deep Neural Networks , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Yang Wang,et al.  Human Action Recognition by Semilatent Topic Models , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  George Kachergis,et al.  A continuous-time neural model for sequential action , 2014, Philosophical Transactions of the Royal Society B: Biological Sciences.

[44]  Brian Scassellati,et al.  Data-Driven Model of Nonverbal Behavior for Socially Assistive Human-Robot Interactions , 2014, ICMI.

[45]  Irfan A. Essa,et al.  Expectation grammars: leveraging high-level expectations for activity recognition , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[46]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[47]  Sean Andrist,et al.  Look Like Me: Matching Robot Personality via Gaze to Increase Motivation , 2015, CHI.

[48]  J. Gregory Trafton,et al.  Accommodating Human Variability in Human-Robot Teams through Theory of Mind , 2011, IJCAI.

[49]  Brian Scassellati,et al.  Theory of Mind for a Humanoid Robot , 2002, Auton. Robots.

[50]  Chen Liang,et al.  Constructing Hierarchical Concepts via Analogical Generalization , 2014, CogSci.

[51]  Chao Ou-Yang,et al.  Petri-net integration - An approach to support multi-agent process mining , 2011, Expert Syst. Appl..

[52]  François Brémond,et al.  Automatic Video Interpretation: A Novel Algorithm for Temporal Scenario Recognition , 2003, IJCAI.

[53]  David W. Aha,et al.  Goal Reasoning for an Autonomous Squad Member , 2015 .

[54]  Prügel-BennettAdam,et al.  Training HMM structure with genetic algorithm for biological sequence analysis , 2004 .

[55]  Jianxin Wu Hidden Markov model , 2018 .

[56]  Rama Chellappa,et al.  Machine Recognition of Human Activities: A Survey , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[57]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[58]  Eric Horvitz,et al.  Layered representations for human activity recognition , 2002, Proceedings. Fourth IEEE International Conference on Multimodal Interfaces.

[59]  Julie A. Shah,et al.  Fast target prediction of human reaching motion for cooperative human-robot manipulation tasks using time series classification , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[60]  Ehud Rivlin,et al.  Video Event Modeling and Recognition in Generalized Stochastic Petri Nets , 2010, IEEE Transactions on Circuits and Systems for Video Technology.

[61]  Ladislau Bölöni,et al.  Analyzing Team Actions with Cascading HMM , 2009, FLAIRS.

[62]  Tadao Murata,et al.  Petri nets: Properties, analysis and applications , 1989, Proc. IEEE.

[63]  Shlomo Zilberstein,et al.  Temporal and Object Relations in Unsupervised Plan and Activity Recognition , 2015, AAAI Fall Symposia.

[64]  Rama Chellappa,et al.  A Constrained Probabilistic Petri Net Framework for Human Activity Detection in Video* , 2008, IEEE Transactions on Multimedia.

[65]  D. Gentner Structure‐Mapping: A Theoretical Framework for Analogy* , 1983 .

[66]  C. Raymond Perrault,et al.  A Plan-Based Analysis of Indirect Speech Act , 1980, CL.

[67]  Svetha Venkatesh,et al.  Learning Hierarchical Hidden Markov Models with General State Hierarchy , 2004, AAAI.

[68]  Robert P. Goldman,et al.  A Bayesian Model of Plan Recognition , 1993, Artif. Intell..

[69]  Siddhartha S. Srinivasa,et al.  Legibility and predictability of robot motion , 2013, 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[70]  B. Risfic,et al.  Beyond the kalman filter - Book Review , 2004, IEEE Aerospace and Electronic Systems Magazine.

[71]  Qiang Yang,et al.  Boosting for transfer learning , 2007, ICML '07.

[72]  Shaogang Gong,et al.  Recognition of group activities using dynamic probabilistic networks , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[73]  Hung Hai Bui,et al.  A General Model for Online Probabilistic Plan Recognition , 2003, IJCAI.

[74]  Larry S. Davis,et al.  Representation and Recognition of Events in Surveillance Video Using Petri Nets , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[75]  Takaki Makino,et al.  Apprenticeship Learning for Model Parameters of Partially Observable Environments , 2012, ICML.

[76]  Thomas Serre,et al.  The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[77]  Yichuan Tang,et al.  Deep Learning using Linear Support Vector Machines , 2013, 1306.0239.

[78]  J. Gregory Trafton,et al.  Using Simulations to Model Shared Mental Models , 2007 .

[79]  Zoubin Ghahramani,et al.  An Introduction to Hidden Markov Models and Bayesian Networks , 2001, Int. J. Pattern Recognit. Artif. Intell..

[80]  J. Gregory Trafton,et al.  Building and Verifying a Predictive Model of Interruption Resumption , 2012, Proceedings of the IEEE.

[81]  Yan Meng,et al.  Human activity recognition in video using a hierarchical probabilistic latent model , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[82]  Terrence Fong,et al.  Collaboration, Dialogue, and Human-Robot Interaction , 2001 .

[83]  Andreas Stolcke,et al.  Hidden Markov Model} Induction by Bayesian Model Merging , 1992, NIPS.

[84]  Bernt Schiele,et al.  Discovery of activity patterns using topic models , 2008 .

[85]  Marc B. Vilain,et al.  Getting Serious about Parsing Plans : a Grammatical Analysis of Plan Recognition , 1990 .

[86]  Robert Platt,et al.  Extracting User Intent in Mixed Initiative Teleoperator Control , 2004 .

[87]  Martha W. Alibali,et al.  Language, gesture, action! A test of the Gesture as Simulated Action framework , 2010 .

[88]  Aude Billard,et al.  On Learning, Representing, and Generalizing a Task in a Humanoid Robot , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[89]  Eric Horvitz,et al.  A Comparison of HMMs and Dynamic Bayesian Networks for Recognizing Office Activities , 2005, User Modeling.

[90]  John R. Anderson How Can the Human Mind Occur in the Physical Universe , 2007 .

[91]  Autumn B. Hostetter,et al.  Visible embodiment: Gestures as simulated action , 2008, Psychonomic bulletin & review.

[92]  Greg Mori,et al.  IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL., NO. 1 Human Action Recognition by Semi-Latent Topic Models , 2022 .

[93]  R. Weale Vision. A Computational Investigation Into the Human Representation and Processing of Visual Information. David Marr , 1983 .

[94]  J. Gregory Trafton,et al.  ACT-R/E , 2013, HRI 2013.

[95]  Yoichi Sato,et al.  Recovering the Basic Structure of Human Activities from a Video-Based Symbol String , 2007, 2007 IEEE Workshop on Motion and Video Computing (WMVC'07).

[96]  Michael L. Littman,et al.  Activity Recognition from Accelerometer Data , 2005, AAAI.

[97]  Terrence C. Stewart,et al.  Building Production Systems with Realistic Spiking Neurons , 2008 .

[98]  Sarit Kraus,et al.  The Evolution of Sharedplans , 1999 .

[99]  Rüdiger Dillmann,et al.  Markerless human motion tracking with a flexible model and appearance learning , 2009, 2009 IEEE International Conference on Robotics and Automation.

[100]  J. Tenenbaum,et al.  Probabilistic models of cognition: exploring representations and inductive biases , 2010, Trends in Cognitive Sciences.

[101]  Siddhartha S. Srinivasa,et al.  Formalizing Assistive Teleoperation , 2012, Robotics: Science and Systems.

[102]  Hila Zarosim,et al.  Fast and Complete Symbolic Plan Recognition: Allowing for Duration, Interleaved Execution, and Lossy Observations , 2005 .

[103]  Stefan Kopp,et al.  MODELING THE PRODUCTION OF COVERBAL ICONIC GESTURES BY LEARNING BAYESIAN DECISION NETWORKS , 2010, Appl. Artif. Intell..

[104]  Kenneth D. Forbus,et al.  MAC/FAC: A Model of Similarity-Based Retrieval , 1995, Cogn. Sci..

[105]  Ana Paiva,et al.  Detecting Engagement in HRI: An Exploration of Social and Task-Based Context , 2012, 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing.

[106]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[107]  Hector Geffner,et al.  Goal Recognition over POMDPs: Inferring the Intention of a POMDP Agent , 2011, IJCAI.

[108]  Raj M. Ratwani,et al.  A memory for goals model of sequence errors , 2011, Cognitive Systems Research.

[109]  Roland Siegwart,et al.  Robot learning from demonstration , 2004, Robotics Auton. Syst..

[110]  Jay Earley,et al.  An efficient context-free parsing algorithm , 1970, Commun. ACM.

[111]  Michael P. Wellman,et al.  Probabilistic State-Dependent Grammars for Plan Recognition , 2000, UAI.

[112]  Vladimir Pavlovic,et al.  A dynamic Bayesian network approach to figure tracking using learned dynamic models , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[113]  D. Kahneman Thinking, Fast and Slow , 2011 .

[114]  Stuart J. Russell,et al.  Dynamic bayesian networks: representation, inference and learning , 2002 .

[115]  Thomas L. Griffiths,et al.  Integrating Topics and Syntax , 2004, NIPS.

[116]  Illah R. Nourbakhsh,et al.  A survey of socially interactive robots , 2003, Robotics Auton. Syst..

[117]  John R. Anderson A Spreading Activation Theory of Memory , 1988 .

[118]  Branko Ristic,et al.  Beyond the Kalman Filter: Particle Filters for Tracking Applications , 2004 .

[119]  Bruce D'Ambrosio,et al.  Proceedings of the Eighth international conference on Uncertainty in artificial intelligence , 1992 .

[120]  David W. Aha,et al.  Increasing the Runtime Speed of Case-Based Plan Recognition , 2015, FLAIRS.

[121]  Susan M. Wagner,et al.  Probing the Mental Representation of Gesture: Is Handwaving Spatial?. , 2004 .

[122]  Eric Horvitz,et al.  Dynamic Network Models for Forecasting , 1992, UAI.

[123]  Monica N. Nicolescu,et al.  Understanding human intentions via Hidden Markov Models in autonomous mobile robots , 2008, 2008 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[124]  Svetha Venkatesh,et al.  Learning and detecting activities from movement trajectories using the hierarchical hidden Markov model , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[125]  K. J. Craik,et al.  The nature of explanation , 1944 .

[126]  Sandra Carberry,et al.  Techniques for Plan Recognition , 2001, User Modeling and User-Adapted Interaction.

[127]  Christian Lebiere,et al.  Unsurpervised Learning in Hybrid Cognitive Architectures , 2012 .

[128]  Kenneth D. Forbus,et al.  Modeling perceptual similarity as analogy resolves the paradox of difference detection , 2009 .

[129]  Jake K. Aggarwal,et al.  Recognition of Composite Human Activities through Context-Free Grammar Based Representation , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[130]  Joshua B. Tenenbaum,et al.  Bayesian Modeling of Human Concept Learning , 1998, NIPS.

[131]  Adam Prügel-Bennett,et al.  Training HMM structure with genetic algorithm for biological sequence analysis , 2004, Bioinform..

[132]  Ilkka Korhonen,et al.  Detection of Daily Activities and Sports With Wearable Sensors in Controlled and Uncontrolled Conditions , 2008, IEEE Transactions on Information Technology in Biomedicine.

[133]  Dana Kulic,et al.  Representability of human motions by factorial hidden Markov models , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[134]  Gwenn Englebienne,et al.  Behavior analysis of elderly using topic models , 2014, Pervasive Mob. Comput..

[135]  Odest Chadwicke Jenkins,et al.  Tracking human motion and actions for interactive robots , 2007, 2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[136]  Jake K. Aggarwal,et al.  Semantic Representation and Recognition of Continued and Recursive Human Activities , 2009, International Journal of Computer Vision.

[137]  Takayuki Kanda,et al.  How do people walk side-by-side? — Using a computational model of human behavior for a social robot , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[138]  Andrew McCallum,et al.  Information Extraction with HMM Structures Learned by Stochastic Optimization , 2000, AAAI/IAAI.

[139]  Henry A. Kautz,et al.  Generalized Plan Recognition , 1986, AAAI.

[140]  Brian Charles Williams,et al.  Concurrent Plan Recognition and Execution for Human-Robot Teams , 2014, ICAPS.

[141]  Allison M. Okamura,et al.  Recognition of operator motions for real-time assistance using virtual fixtures , 2003, 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2003. HAPTICS 2003. Proceedings..

[142]  S. Eddy Hidden Markov models. , 1996, Current opinion in structural biology.

[143]  Ramakant Nevatia,et al.  Hierarchical Language-based Representation of Events in Video Streams , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.