A Bayesian Computer Vision System for Modeling Human Interactions

We describe a real-time computer vision and machine learning system for modeling and recognizing human behaviors in a visual surveillance task. The system deals in particularly with detecting when interactions between people occur and classifying the type of interaction. Examples of interesting interaction behaviors include following another person, altering one's path to meet another, and so forth. Our system combines top-down with bottom-up information in a closed feedback loop, with both components employing a statistical Bayesian approach. We propose and compare two different state-based learning architectures, namely, HMMs and CHMMs for modeling behaviors and interactions. Finally, a synthetic "Alife-style" training system is used to develop flexible prior models for recognizing human interactions. We demonstrate the ability to use these a priori models to accurately classify real human behaviors and interactions with no additional tuning or training.

[1]  H. Buxton,et al.  Advanced visual surveillance using Bayesian networks , 1997 .

[2]  Alex Pentland,et al.  Graphical Models for Recognizing Human Interactions , 1998, NIPS.

[3]  Lawrence R. Rabiner,et al.  A tutorial on Hidden Markov Models , 1986 .

[4]  Aaron F. Bobick,et al.  Computers Seeing Action , 1996, BMVC.

[5]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[6]  Anthony G. Cohn,et al.  Building qualitative event models automatically from visual input , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[7]  Alex Pentland,et al.  Pfinder: Real-Time Tracking of the Human Body , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  A. Pentland,et al.  Blob - An unsupervised clustering approach to spatial preprocessing of MSS imagery , 1977 .

[9]  Alex Pentland,et al.  Probabilistic visual learning for object detection , 1995, Proceedings of IEEE International Conference on Computer Vision.

[10]  Alex Pentland,et al.  A Bayesian Computer Vision System for Modeling Human Interaction , 1999, ICVS.

[11]  Hans-Hellmut Nagel,et al.  From image sequences towards conceptual descriptions , 1988, Image Vis. Comput..

[12]  Michael I. Jordan,et al.  Hidden Markov Decision Trees , 1996, NIPS.

[13]  BuntineWray.  A Guide to the Literature on Learning Probabilistic Networks from Data , 1996 .

[14]  Alex Pentland,et al.  Pfinder: real-time tracking of the human body , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[15]  Alex Pentland,et al.  Modeling and Prediction of Human Behavior , 1999, Neural Computation.

[16]  David Heckerman,et al.  Causal independence for probability assessment and inference using Bayesian networks , 1996, IEEE Trans. Syst. Man Cybern. Part A.

[17]  Wray L. Buntine.  Operations for Learning with Graphical Models , 1994, J. Artif. Intell. Res..

[18]  Jitendra Malik,et al.  Automatic Symbolic Traffic Scene Analysis Using Belief Networks , 1994, AAAI.

[19]  David G. Stork,et al.  Invited Speech: Speechreading: An Overview of Image Processing, Feature Extraction, Sensory Intergration and Pattern Recognition Techiques , 1996 .

[20]  Matthew Brand,et al.  Coupled hidden Markov models for modeling interacting processes , 1997 .

[21]  Geoffrey E. Hinton,et al.  Mean field networks that learn to discriminate temporally distorted strings , 1991 .

[22]  Robert C. Bolles,et al.  The Representation Space Paradigm of Concurrent Evolving Object Descriptions , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Alex Pentland,et al.  Active gesture recognition using partially observable Markov decision processes , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[24]  Alex Pentland,et al.  LAFTER: lips and face real time tracker , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[25]  David G. Stork,et al.  Speechreading: an overview of image processing, feature extraction, sensory integration and pattern recognition techniques , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[26]  Wray L. Buntine.  A Guide to the Literature on Learning Probabilistic Networks from Data , 1996, IEEE Trans. Knowl. Data Eng..

[27]  Michael I. Jordan,et al.  Probabilistic Independence Networks for Hidden Markov Probability Models , 1997, Neural Computation.

[28]  Alex Pentland,et al.  A synthetic agent system for Bayesian modeling of human interactions , 1999, AGENTS '99.

[29]  Alex Pentland,et al.  Towards perceptual intelligence: statistical modeling of human individual and interactive behaviors , 2000 .

[30]  Michael I. Jordan,et al.  Boltzmann Chains and Hidden Markov Models , 1994, NIPS.

[31]  Alex Pentland.  Classification by Clustering , 1976 .

[32]  Alex Pentland,et al.  Coupled hidden Markov models for complex action recognition , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.