Socially intelligent surveillance and monitoring: Analysing social dimensions of physical space

In general terms, surveillance and monitoring technologies aim at understanding what people do in a given environment, whether this means to ensure the safety of workers on the factory floor, to detect crimes occurring in indoor or outdoor settings, or to monitor the flow of large crowds through public spaces. However, surveillance and monitoring technologies rarely consider that they analyze human behavior, a phenomenon subject to principles and laws rigorous enough to produce stable and predictable patterns corresponding to social, affective, and psychological phenomena. On the other hand, these phenomena are the subject of other computing domains, in particular Social Signal Processing and Affective Computing, that typically neglect scenarios relevant to surveillance and monitoring technologies, especially when it comes to social and affective dimensions of space in human activities. The goal of this paper is to show that the investigation of the overlapping area between surveillance and monitoring on one side, and Social Signal Processing and Affective Computing on the other side can bring significant progress in both domains and open a number of interesting research perspectives.

[1]  Hélène Laurent,et al.  Review and evaluation of commonly-implemented background subtraction algorithms , 2008, 2008 19th International Conference on Pattern Recognition.

[2]  Alex Pentland,et al.  Modeling and Prediction of Human Behavior , 1999, Neural Computation.

[3]  Ashraf A. Kassim,et al.  Tracking a Variable Number of Human Groups in Video Using Probability Hypothesis Density , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[4]  G. O'Connor.  Small Groups , 1980 .

[5]  Tieniu Tan,et al.  A survey on visual surveillance of object motion and behaviors , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[6]  Azriel Rosenfeld,et al.  Tracking Groups of People , 2000, Comput. Vis. Image Underst..

[7]  Sumit Basu,et al.  Learning Human Interactions w ith the Influence Model , 2001, NIPS 2001.

[8]  Aaron F. Bobick,et al.  Video surveillance of interactions , 1999, Proceedings Second IEEE Workshop on Visual Surveillance (VS'99) (Cat. No.98-89223).

[9]  Hilary Buxton,et al.  Learning and understanding dynamic scene activity: a review , 2003, Image Vis. Comput..

[10]  M. Knapp,et al.  Nonverbal communication in human interaction , 1972 .

[11]  Ramakant Nevatia,et al.  Large-scale event detection using semi-hidden Markov models , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[12]  Ramakant Nevatia,et al.  Hierarchical Multi-channel Hidden Semi Markov Models , 2007, IJCAI.

[13]  Sumit Basu,et al.  Modeling Conversational Dynamics as a Mixed-Memory Markov Process , 2004, NIPS.

[14]  R. Belk.  Situational Variables and Consumer Behavior , 1975 .

[15]  P. Ball.  Critical Mass: How One Thing Leads to Another , 2004 .

[16]  Alex Pentland,et al.  Social signals, their function, and automatic analysis: a survey , 2008, ICMI '08.

[17]  Daniel Gatica-Perez,et al.  Automatic nonverbal analysis of social interaction in small groups: A review , 2009, Image Vis. Comput..

[18]  Alex Pentland,et al.  Coupled hidden Markov models for complex action recognition , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[19]  Serge J. Belongie,et al.  Behavior recognition via sparse spatio-temporal features , 2005, 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance.

[20]  Mubarak Shah,et al.  Abnormal crowd behavior detection using social force model , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[21]  A. Kendon.  Conducting Interaction: Patterns of Behavior in Focused Encounters , 1990 .

[22]  Gregory D. Hager,et al.  Probabilistic data association methods in visual tracking of groups , 2004, CVPR 2004.

[23]  Monique Thonnat,et al.  Tracking Groups of People for Video Surveillance , 2002 .

[24]  A. Scheflen,et al.  Body Language and the Social Order , 1973 .

[25]  Maja Pantic,et al.  Social signal processing: Survey of an emerging domain , 2009, Image Vis. Comput..

[26]  Sergio A. Velastin,et al.  Intelligent distributed surveillance systems: a review , 2005 .

[27]  J. C. Matter.  Video motion detection for physical security applications , 1990 .

[28]  Shyamsundar Rajaram,et al.  Human Activity Recognition Using Multidimensional Indexing , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[29]  Deborah J. Bickford.  The importance of physical space in creating supportive learning environments , 2002 .

[30]  David C. Hogg,et al.  Learning Variable-Length Markov Models of Behavior , 2001, Comput. Vis. Image Underst..

[31]  P. L. Venetianer,et al.  The evolution of video surveillance: an overview , 2008, Machine Vision and Applications.

[32]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[33]  Sergio A. Velastin,et al.  People tracking in surveillance applications , 2006, Image Vis. Comput..

[34]  Massimo Piccardi,et al.  Background subtraction techniques: a review , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[35]  Juan Carlos Niebles,et al.  Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words , 2006, BMVC.

[36]  Sze Kim Pang,et al.  Models and Algorithms for Detection and Tracking of Coordinated Groups , 2008, 2007 5th International Symposium on Image and Signal Processing and Analysis.

[37]  Laura K. Guerrero,et al.  The nonverbal communication reader : classic and contemporary readings , 2008 .

[38]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[39]  Vassilios Morellas,et al.  Counting People in Groups , 2009, 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance.

[40]  Mubarak Shah,et al.  Detecting global motion patterns in complex videos , 2008, 2008 19th International Conference on Pattern Recognition.

[41]  E. Hall,et al.  The Hidden Dimension , 1970 .

[42]  Samy Bengio,et al.  Automatic analysis of multimodal group actions in meetings , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Sergio A. Velastin,et al.  Intelligent distributed video surveillance systems , 2006 .

[44]  W. Eric L. Grimson,et al.  Learning Patterns of Activity Using Real-Time Tracking , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  Badrinath Roysam,et al.  Image change detection algorithms: a systematic survey , 2005, IEEE Transactions on Image Processing.

[46]  Larry S. Davis,et al.  W4: Real-Time Surveillance of People and Their Activities , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[47]  P. Shalit.  The Silent Language , 1964 .

[48]  Shaogang Gong,et al.  Associating Groups of People , 2009, BMVC.

[49]  Thomas Mauthner,et al.  Robust tracking of spatial related components , 2008, 2008 19th International Conference on Pattern Recognition.

[50]  Alex Pentland,et al.  Towards Measuring Human Interactions in Conversational Settings , 2001 .

[51]  Alex Pentland,et al.  Pfinder: Real-Time Tracking of the Human Body , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  Alessandro Perina,et al.  Person re-identification by symmetry-driven accumulation of local features , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[53]  Alex Pentland,et al.  Social signal processing: state-of-the-art and future perspectives of an emerging domain , 2008, ACM Multimedia.

[54]  A F Bobick,et al.  Movement, activity and action: the role of knowledge in the perception of motion. , 1997, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[55]  Mubarak Shah,et al.  A Lagrangian Particle Dynamics Approach for Crowd Flow Segmentation and Stability Analysis , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[56]  Zhihong Zeng,et al.  A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Vittorio Murino,et al.  Towards a Subject-Centered Analysis for Automated Video Surveillance , 2009, ICIAP.

[58]  Svetha Venkatesh,et al.  Activity recognition and abnormality detection with the switching hidden semi-Markov model , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[59]  Ramakant Nevatia,et al.  VERL: An Ontology Framework for Representing and Annotating Video Events , 2005, IEEE Multim..

[60]  James W. Davis,et al.  The Recognition of Human Movement Using Temporal Templates , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[61]  Rama Chellappa,et al.  Recognition of Humans and Their Activities Using Video , 2005, Recognition of Humans and Their Activities Using Video.

[62]  Alex Pentland,et al.  Graphical Models for Recognizing Human Interactions , 1998, NIPS.

[63]  W. Eric L. Grimson,et al.  Using adaptive tracking to classify and monitor activities in a site , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[64]  Ian D. Reid,et al.  A general method for human activity recognition in video , 2006, Comput. Vis. Image Underst..

[65]  D. McFarland.  Respiratory markers of conversational interaction. , 2001, Journal of speech, language, and hearing research : JSLHR.

[66]  Jorge S. Marques,et al.  Tracking Groups of Pedestrians in Video Sequences , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[67]  Rama Chellappa,et al.  Machine Recognition of Human Activities: A Survey , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[68]  Carlo Drioli,et al.  Auditory dialog analysis and understanding by generative modelling of interactional dynamics , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[69]  Yuan F. Zheng,et al.  Tracking a group of highly correlated targets , 2009, 2009 16th IEEE International Conference on Image Processing (ICIP).