Recognizing Behavior in Hand-Eye Coordination Patterns

Modeling human behavior is important for the design of robots as well as human-computer interfaces that use humanoid avatars. Constructive models have been built, but they have not captured all of the detailed structure of human behavior such as the moment-to-moment deployment and coordination of hand, head and eye gaze used in complex tasks. We show how this data from human subjects performing a task can be used to program a dynamic Bayes network (DBN) which in turn can be used to recognize new performance instances. As a specific demonstration we show that the steps in a complex activity such as sandwich making can be recognized by a DBN in real time.

[1]  A. L. Yarbus,et al.  Eye Movements and Vision , 1967, Springer US.

[2]  Andrew Liu,et al.  MODELING AND PREDICTION OF HUMAN DRIVER BEHAVIOR , 2001 .

[3]  Stefan Schaal,et al.  Computational approaches to motor learning by imitation. , 2003, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[4]  Norman I. Badler,et al.  Interactive real-time articulated figure manipulation using multiple kinematic constraints , 1990, I3D '90.

[5]  P. Subramanian.  Active Vision: The Psychology of Looking and Seeing , 2006 .

[6]  L. Stark,et al.  Scanpaths in Eye Movements during Pattern Perception , 1971, Science.

[7]  Cynthia Breazeal,et al.  Learning From and About Others: Towards Using Imitation to Bootstrap the Social Understanding of Others by Robots , 2005, Artificial Life.

[8]  Henry A. Kautz,et al.  Inferring activities from interactions with objects , 2004, IEEE Pervasive Computing.

[9]  Michael J. Swain,et al.  Color indexing , 1991, International Journal of Computer Vision.

[10]  A. Schofield,et al.  Asymmetric transfer of the dynamic motion aftereffect between first- and second-order cues and among different second-order cues. , 2007, Journal of vision.

[11]  Tamim Asfour,et al.  Imitation Learning of Dual-Arm Manipulation Tasks in Humanoid Robots , 2006, 2006 6th IEEE-RAS International Conference on Humanoid Robots.

[12]  Rajesh P. N. Rao,et al.  A Cognitive Model of Imitative Development in Humans and Machines , 2007, Int. J. Humanoid Robotics.

[13]  B. Tatler,et al.  Yarbus, eye movements, and vision , 2010, i-Perception.

[14]  Dana H. Ballard,et al.  Animate Vision , 1991, Artif. Intell..

[15]  Stuart J. Russell,et al.  Dynamic bayesian networks: representation, inference and learning , 2002 .

[16]  Stefan Schaal,et al.  Is imitation learning the route to humanoid robots? , 1999, Trends in Cognitive Sciences.

[17]  Dana H. Ballard,et al.  Routine based models of anticipation in natural behaviors , 2005, AAAI 2005.

[18]  Chen Yu,et al.  Learning to recognize human action sequences , 2002, Proceedings 2nd International Conference on Development and Learning. ICDL 2002.

[19]  A. L. I︠A︡rbus.  Eye Movements and Vision , 1967 .

[20]  Alex Pentland,et al.  Towards perceptual intelligence: statistical modeling of human individual and interactive behaviors , 2000 .

[21]  Odest Chadwicke Jenkins,et al.  Interactive Human Pose and Action Recognition Using Dynamical Motion Primitives , 2007, Int. J. Humanoid Robotics.

[22]  José Santos-Victor,et al.  Visual learning by imitation with motor representations , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[23]  Michael C. Horsch,et al.  Dynamic Bayesian networks , 1990 .

[24]  Eric B. Baum,et al.  What is thought? , 2003 .

[25]  Dana H. Ballard,et al.  Modeling embodied visual behaviors , 2007, TAP.

[26]  Mary M Hayhoe,et al.  Task and context determine where you look. , 2016, Journal of vision.

[27]  Norman I. Badler.  Virtual beings , 2001, CACM.