A review on vision techniques applied to Human Behaviour Analysis for Ambient-Assisted Living

Human Behaviour Analysis (HBA) is more and more being of interest for computer vision and artificial intelligence researchers. Its main application areas, like Video Surveillance and Ambient-Assisted Living (AAL), have been in great demand in recent years. This paper provides a review on HBA for AAL and ageing in place purposes focusing specially on vision techniques. First, a clearly defined taxonomy is presented in order to classify the reviewed works, which are consequently presented following a bottom-up abstraction and complexity order. At the motion level, pose and gaze estimation as well as basic human movement recognition are covered. Next, the mainly used action and activity recognition approaches are presented with examples of recent research works. Increasing the degree of semantics and the time interval involved in the HBA, finally the behaviour level is reached. Furthermore, useful tools and datasets are analysed in order to provide help for initiating projects.

[1]  Neil Robertson,et al.  Behaviour Recognition and Explanation for Video Surveillance , 2006 .

[2]  Andrew McCallum,et al.  An Introduction to Conditional Random Fields for Relational Learning , 2007 .

[3]  Stephen J. McKenna,et al.  Activity summarisation and fall detection in a supportive home environment , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[4]  Nicholas R. Howe,et al.  Silhouette Lookup for Automatic Pose Tracking , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[5]  Chris D. Nugent,et al.  Evidential fusion of sensor data for activity recognition in smart homes , 2009, Pervasive Mob. Comput..

[6]  Christopher Joseph Pal,et al.  Activity recognition using the velocity histories of tracked keypoints , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[7]  Thomas B. Moeslund,et al.  A Survey of Computer Vision-Based Human Motion Capture , 2001, Comput. Vis. Image Underst..

[8]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Michael Beetz,et al.  Accurate Human Motion Capture Using an Ergonomics-Based Anthropometric Human Model , 2008, AMDO.

[10]  Ling Bao,et al.  Activity Recognition from User-Annotated Acceleration Data , 2004, Pervasive.

[11]  Andrew Zisserman,et al.  "Here's looking at you, kid". Detecting people looking at each other in videos , 2011, BMVC.

[12]  Ivan Laptev,et al.  On Space-Time Interest Points , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[13]  Peter H. Veltink,et al.  Ambulatory Position and Orientation Tracking Fusing Magnetic and Inertial Sensing , 2007, IEEE Transactions on Biomedical Engineering.

[14]  Mohan M. Trivedi,et al.  Human Body Model Acquisition and Tracking Using Voxel Data , 2003, International Journal of Computer Vision.

[15]  François Brémond,et al.  Monitoring Activities of Daily Living (ADLs) of Elderly Based on 3D Key Human Postures , 2009, ICVW.

[16]  Jenny Benois-Pineau,et al.  Human Daily Activities Indexing in Videos from Wearable Cameras for Monitoring of Patients with Dementia Diseases , 2010, 2010 20th International Conference on Pattern Recognition.

[17]  Jake K. Aggarwal,et al.  Spatio-temporal relationship match: Video structure comparison for recognition of complex human activities , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[18]  Bernt Schiele,et al.  Pictorial structures revisited: People detection and articulated pose estimation , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  John A. Stankovic,et al.  Context-aware wireless sensor networks for assisted living and residential monitoring , 2008, IEEE Network.

[20]  Michael Beetz,et al.  EYEWATCHME—3D Hand and object tracking for inside out activity analysis , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[21]  Kent Larson,et al.  Activity Recognition in the Home Using Simple and Ubiquitous Sensors , 2004, Pervasive.

[22]  Andreas Savvides,et al.  The BehaviorScope framework for enabling ambient assisted living , 2010, Personal and Ubiquitous Computing.

[23]  A. Mihailidis,et al.  The COACH prompting system to assist older adults with dementia through handwashing: An efficacy study , 2008, BMC geriatrics.

[24]  Huosheng Hu,et al.  Integration of Vision and Inertial Sensors for 3D Arm Motion Tracking in Home-based Rehabilitation , 2007, Int. J. Robotics Res..

[25]  Hadi Aliakbarpour,et al.  Using concurrent Hidden Markov Models to analyse human behaviours in a smart home environment , 2011, WIAMIS 2011.

[26]  Lijun Yin,et al.  Viewing direction estimation based on 3D eyeball construction for HRI , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[27]  Alan M. McIvor,et al.  Background Subtraction Techniques , 2000 .

[28]  Adrian Hilton,et al.  A survey of advances in vision-based human motion capture and analysis , 2006, Comput. Vis. Image Underst..

[29]  Gary M. Weiss,et al.  Activity recognition using cell phone accelerometers , 2011, SKDD.

[30]  Dariu Gavrila,et al.  The Visual Analysis of Human Movement: A Survey , 1999, Comput. Vis. Image Underst..

[31]  Svetha Venkatesh,et al.  Activity recognition and abnormality detection with the switching hidden semi-Markov model , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[32]  V. Ramasubramanian,et al.  Towards fast, view-invariant human action recognition , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[33]  Alex Pentland,et al.  A Bayesian Computer Vision System for Modeling Human Interactions , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[34]  Moritz Tenorth,et al.  Towards Automated Models of Activities of Daily Life , 2009 .

[35]  Shyamsundar Rajaram,et al.  Human Activity Recognition Using Multidimensional Indexing , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[36]  Ankur Agarwal,et al.  3D human pose from silhouettes by relevance vector regression , 2004, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004..

[37]  Cordelia Schmid,et al.  Learning realistic human actions from movies , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  Dorothy Ndedi Monekosso,et al.  Behavior Analysis for Assisted Living , 2010, IEEE Transactions on Automation Science and Engineering.

[39]  Josef Hallberg,et al.  homeML - An Open Standard for the Exchange of Data Within Smart Environments , 2007, ICOST.

[40]  Rémi Ronfard,et al.  Free viewpoint action recognition using motion history volumes , 2006, Comput. Vis. Image Underst..

[41]  Nicu Sebe,et al.  Multimodal Human Computer Interaction: A Survey , 2005, ICCV-HCI.

[42]  Farrah Wong,et al.  Performance of Histogram-Based Skin Colour Segmentation for Arms Detection in Human Motion Analysis Application , 2009 .

[43]  Matthai Philipose,et al.  Egocentric recognition of handled objects: Benchmark and analysis , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[44]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[45]  Ben Taskar,et al.  An Introduction to Conditional Random Fields for Relational Learning , 2007 .

[46]  Arthur P. Dempster,et al.  A Generalization of Bayesian Inference , 1968, Classic Works of the Dempster-Shafer Theory of Belief Functions.

[47]  Montse Pardàs,et al.  Multimodal real-time focus of attention estimation in SmartRooms , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[48]  T. Poggio,et al.  Direction Estimation of Pedestrian from Images , 2003 .

[49]  Martial Hebert,et al.  Temporal segmentation and activity classification from first-person sensing , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[50]  Andreas Seidl,et al.  Ramsis - The Leading Cad Tool for Ergonomic Analysis of Vehicles , 2007, HCI.

[51]  Tamara L. Hayes,et al.  An approach for deriving continuous health assessment indicators from in-home sensor data , 2008 .

[52]  Jitendra Malik,et al.  Shape Context: A New Descriptor for Shape Matching and Object Recognition , 2000, NIPS.

[53]  Honghai Liu,et al.  Visual-Based View-Invariant Human Motion Analysis: A Review , 2008, KES.

[54]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[55]  M. Brady,et al.  Scale Saliency: a novel approach to salient feature and scale selection , 2003 .

[56]  Billur Barshan,et al.  Human Activity Recognition Using Inertial/Magnetic Sensor Units , 2010, HBU.

[57]  Maja Pantic,et al.  An implicit spatiotemporal shape model for human activity localization and recognition , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[58]  Jessica K. Hodgins,et al.  Guide to the Carnegie Mellon University Multimodal Activity (CMU-MMAC) Database , 2008 .

[59]  Ramakant Nevatia,et al.  Single View Human Action Recognition using Key Pose Matching and Viterbi Path Searching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[60]  Andrew Zisserman,et al.  Progressive search space reduction for human pose estimation , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[61]  Pau-Choo Chung,et al.  An Interaction-Embedded HMM Framework for Human Behavior Understanding: With Nursing Environments as Examples , 2010, IEEE Transactions on Information Technology in Biomedicine.

[62]  Jerry L. Prince,et al.  Snakes, shapes, and gradient vector flow , 1998, IEEE Trans. Image Process..

[63]  Andreas P. Schmidt,et al.  SOPRANO – An extensible , open AAL platform for elderly people based on semantical contracts 1 , 2008 .

[64]  James M. Rehg,et al.  A Scalable Approach to Activity Recognition based on Object Use , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[65]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[66]  Vangelis Metsis,et al.  Abnormal human behavioral pattern detection in assisted living environments , 2010, PETRA '10.

[67]  Barbara Caputo,et al.  Recognizing human actions: a local SVM approach , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[68]  Vittorio Ferrari,et al.  Better Appearance Models for Pictorial Structures , 2009, BMVC.

[69]  Walterio W. Mayol-Cuevas,et al.  High level activity recognition using low resolution wearable vision , 2009, 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[70]  Kiyoharu Aizawa,et al.  Tracking of humans and estimation of body/head orientation from top-view single camera for visual focus of attention analysis , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[71]  James W. Davis,et al.  The Recognition of Human Movement Using Temporal Templates , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[72]  Birgitta König-Ries,et al.  openAAL 1the open source middleware for ambient-assisted living ( AAL ) , 2010 .

[73]  W. Cleghorn,et al.  The development of a computerized cueing device to help people with dementia to be more independent , 2001 .

[74]  Subhransu Maji,et al.  Detecting People Using Mutually Consistent Poselet Activations , 2010, ECCV.

[75]  Rainer Stiefelhagen,et al.  Multi-view Based Estimation of Human Upper-Body Orientation , 2010, 2010 20th International Conference on Pattern Recognition.

[76]  Ramakant Nevatia,et al.  Detection of multiple, partially occluded humans in a single image by Bayesian combination of edgelet part detectors , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[77]  Pau-Choo Chung,et al.  A daily behavior enabled hidden Markov model for human behavior understanding , 2008, Pattern Recognit..

[78]  Zicheng Liu,et al.  Expandable Data-Driven Graphical Modeling of Human Actions Based on Salient Postures , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[79]  Xi Chen,et al.  Activity Analysis, Summarization, and Visualization for Indoor Human Activity Monitoring , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[80]  Ben Taskar,et al.  Cascaded Models for Articulated Pose Estimation , 2010, ECCV.

[81]  S. Govindarajulu,et al.  A Comparison of SIFT, PCA-SIFT and SURF , 2012 .

[82]  Josephine Sullivan,et al.  Contextual Features for Head Pose Estimation in Football Games , 2010, 2010 20th International Conference on Pattern Recognition.

[83]  Gaetano Borriello,et al.  A Practical Approach to Recognizing Physical Activities , 2006, Pervasive.

[84]  Rama Chellappa,et al.  Machine Recognition of Human Activities: A Survey , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[85]  Allen Y. Yang,et al.  Distributed segmentation and classification of human actions using a wearable motion sensor network , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[86]  Svetha Venkatesh,et al.  Learning and detecting activities from movement trajectories using the hierarchical hidden Markov model , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[87]  Kamiar Aminian,et al.  Ambulatory system for human motion analysis using a kinematic sensor: monitoring of daily physical activity in the elderly , 2003, IEEE Transactions on Biomedical Engineering.

[88]  Takashi Omori,et al.  Detection of unusual human behavior in intelligent house , 2002, Proceedings of the 12th IEEE Workshop on Neural Networks for Signal Processing.

[89]  Trevor Darrell,et al.  Fast pose estimation with parameter-sensitive hashing , 2003, Proceedings Ninth IEEE International Conference on Computer Vision.

[90]  Hui Cheng,et al.  HO2: A new feature for multi-agent event detection and recognition , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[91]  Mohan M. Trivedi,et al.  Attention estimation by simultaneous observation of viewer and view , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[92]  Daniel P. Siewiorek,et al.  Activity recognition and monitoring using multiple sensors on different body positions , 2006, International Workshop on Wearable and Implantable Body Sensor Networks (BSN'06).

[93]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[94]  Moritz Tenorth,et al.  The TUM Kitchen Data Set of everyday manipulation activities for motion tracking and action recognition , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[95]  Ramakant Nevatia,et al.  Multi-agent event recognition , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[96]  Kentaro Toyama,et al.  Wallflower: principles and practice of background maintenance , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[97]  Hans-Peter Seidel,et al.  Free-viewpoint video of human actors , 2003, ACM Trans. Graph..

[98]  K. Aminian,et al.  Physical activity monitoring based on accelerometry: validation and comparison with video observation , 1999, Medical & Biological Engineering & Computing.

[99]  Taisuke Sato,et al.  Bayesian classification of task-oriented actions based on stochastic context-free grammar , 2006, 7th International Conference on Automatic Face and Gesture Recognition (FGR06).

[100]  Rómer Rosales,et al.  Learning and synthesizing human body motion and posture , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[101]  Mohan M. Trivedi,et al.  Examining the impact of driving style on the predictability and responsiveness of the driver: Real-world and simulator analysis , 2010, 2010 IEEE Intelligent Vehicles Symposium.

[102]  D. Cook,et al.  Monitoring Health by Detecting Drifts and Outliers for a Smart Environment Inhabitant 1 , 2006 .

[103]  Cristian Sminchisescu,et al.  Discriminative density propagation for 3D human motion estimation , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[104]  I. Patras,et al.  Spatiotemporal salient points for visual recognition of human actions , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[105]  Oliver Brdiczka,et al.  Learning Situation Models in a Smart Home , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[106]  Mubarak Shah,et al.  Action MACH a spatio-temporal Maximum Average Correlation Height filter for action recognition , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[107]  Ronen Basri,et al.  Actions as space-time shapes , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[108]  Andrea Passerini,et al.  From On-Going to Complete Activity Recognition Exploiting Related Activities , 2010, HBU.

[109]  Lijun Yin,et al.  Pointing with the eyes: Gaze estimation using a static/active camera system and 3D iris disk model , 2010, 2010 IEEE International Conference on Multimedia and Expo.

[110]  Tieniu Tan,et al.  Recent developments in human motion analysis , 2003, Pattern Recognit..

[111]  Henry A. Kautz,et al.  Privacy-Preserving Recognition of Activities in Daily Living from Multi-view Silhouettes and RFID-based Training , 2008, AAAI Fall Symposium: AI in Eldercare: New Solutions to Old Problems.

[112]  Antonio Criminisi,et al.  Object categorization by learned universal visual dictionary , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[113]  A. Sixsmith,et al.  Monitoring activity patterns and trends of older adults , 2008, 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[114]  Luo Juan,et al.  A comparison of SIFT, PCA-SIFT and SURF , 2009 .

[115]  François Brémond,et al.  Applying 3D human model in a posture recognition system , 2006, Pattern Recognit. Lett..

[116]  Fabien Cardinaux,et al.  Modelling of Behavioural Patterns for Abnormality Detection in the Context of Lifestyle Reassurance , 2008, CIARP.

[117]  Matthew Brand,et al.  Coupled hidden Markov models for modeling interacting processes , 1997 .

[118]  Massimo Piccardi,et al.  Background subtraction techniques: a review , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[119]  Juan Carlos Niebles,et al.  Unsupervised Learning of Human Action Categories Using Spatial-Temporal Words , 2006, BMVC.

[120]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[121]  James M. Keller,et al.  Modeling Human Activity From Voxel Person Using Fuzzy Logic , 2009, IEEE Transactions on Fuzzy Systems.

[122]  Ronald Poppe,et al.  A survey on vision-based human action recognition , 2010, Image Vis. Comput..

[123]  Ronald Poppe,et al.  Vision-based human motion analysis: An overview , 2007, Comput. Vis. Image Underst..

[124]  Tieniu Tan,et al.  A survey on visual surveillance of object motion and behaviors , 2004, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[125]  Hossein Ragheb,et al.  MuHAVi: A Multicamera Human Action Video Dataset for the Evaluation of Action Recognition Methods , 2010, 2010 7th IEEE International Conference on Advanced Video and Signal Based Surveillance.

[126]  Rémi Ronfard,et al.  Action Recognition from Arbitrary Views using 3D Exemplars , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[127]  Alex Mihailidis,et al.  The use of computer vision in an intelligent environment to support aging-in-place, safety, and independence in the home , 2004, IEEE Transactions on Information Technology in Biomedicine.

[128]  Ahmad Lotfi,et al.  Trend Modelling of Elderly Lifestyle within an Occupancy Simulator , 2011, 2011 UkSim 13th International Conference on Computer Modelling and Simulation.

[129]  Ian D. Reid,et al.  A general method for human activity recognition in video , 2006, Comput. Vis. Image Underst..

[130]  Takeo Kanade,et al.  Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[131]  Kent Larson,et al.  Using a Live-In Laboratory for Ubiquitous Computing Research , 2006, Pervasive.

[132]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[133]  Maja Pantic,et al.  B-spline polynomial descriptors for human activity recognition , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[134]  Wanqing Li,et al.  Action recognition based on a bag of 3D points , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.