Sensors, vision and networks: From video surveillance to activity recognition and health monitoring

This paper presents an overview of the state of the art of three different fields with the shared characteristics of making use of a network of sensors, with the possible application of computer vision, signal processing, and machine learning algorithms. Namely, the paper first reports the state of the art and possible future directions for Intelligent Video Surveillance (IVS) applications, by recaping the history of the field in terms of hardware and algorithmic progresses. Then, the existing technologies of Wireless Sensor Networks (WSNs) are compared and described. Their applications to human activity recognition (HAR), both from a single or multiple sensors perspectives, are described and classified, followed by the current research trends and challenges. Finally, recent advances on camera-based health monitoring (including vision-based Ambient Assisted Living and patient monitoring, and camera-based physiological measurements) are described in full details, with the challenges faced.

[1]  Vassilis Athitsos,et al.  A survey on vision-based fall detection , 2015, PETRA.

[2]  Ali Ahmadinia,et al.  Evaluation of LoRa and LoRaWAN for wireless sensor networks , 2016, 2016 IEEE SENSORS.

[3]  J. Geoffrey Chase,et al.  Quantifying agitation in sedated ICU patients using digital imaging , 2004, Comput. Methods Programs Biomed..

[5]  F. Mastik,et al.  Contactless Multiple Wavelength Photoplethysmographic Imaging: A First Step Toward “SpO2 Camera” Technology , 2005, Annals of Biomedical Engineering.

[6]  Andreas Braun,et al.  Low-cost indoor localization using cameras - Evaluating AmbiTrack and its applications in Ambient Assisted Living , 2016, J. Ambient Intell. Smart Environ..

[7]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[8]  Massimo Piccardi,et al.  Background subtraction techniques: a review , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[9]  Petia Radeva,et al.  Toward Storytelling From Visual Lifelogging: An Overview , 2015, IEEE Transactions on Human-Machine Systems.

[10]  François Charpillet,et al.  Measuring frailty and detecting falls for elderly home care using depth camera , 2017, J. Ambient Intell. Smart Environ..

[11]  Stuart J. Russell,et al.  Image Segmentation in Video Sequences: A Probabilistic Approach , 1997, UAI.

[12]  C. Randell,et al.  Context awareness by analysing accelerometer data , 2000, Digest of Papers. Fourth International Symposium on Wearable Computers.

[13]  How-Lung Eng,et al.  Computer vision applications for patients monitoring system , 2012, Fusion.

[14]  Andrea Cavallaro,et al.  Multi-Camera Networks: Principles and Applications , 2009 .

[15]  C. Qian,et al.  Frame-rate Multi-body Tracking for Surveillance , 1998 .

[16]  Xiaogang Wang,et al.  Person Re-Identification by Saliency Learning , 2014 .

[17]  Thomas Watteyne,et al.  Understanding the Limits of LoRaWAN , 2016, IEEE Communications Magazine.

[18]  Bin Liu,et al.  Human daily activity recognition by fusing accelerometer and multi-lead ECG data , 2013, 2013 IEEE International Conference on Signal Processing, Communication and Computing (ICSPCC 2013).

[19]  Alex Pentland,et al.  Pfinder: real-time tracking of the human body , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[20]  Sung Wook Baik,et al.  Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features , 2018, IEEE Access.

[21]  Mohan M. Trivedi,et al.  A Survey of Vision-Based Trajectory Learning and Analysis for Surveillance , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[22]  Westgate Road,et al.  Photoplethysmography and its application in clinical physiological measurement , 2007 .

[23]  Nigel H. Lovell,et al.  Implementation of a real-time human movement classifier using a triaxial accelerometer for ambulatory monitoring , 2006, IEEE Transactions on Information Technology in Biomedicine.

[24]  Antoine Vacavant,et al.  A comprehensive review of background subtraction algorithms evaluated with synthetic and real videos , 2014, Comput. Vis. Image Underst..

[25]  Xiaogang Wang,et al.  Intelligent multi-camera video surveillance: A review , 2013, Pattern Recognit. Lett..

[26]  Jesse Hoey,et al.  Sensor-Based Activity Recognition , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[27]  Zhenyu He,et al.  Activity recognition from accelerometer signals based on Wavelet-AR model , 2010, 2010 IEEE International Conference on Progress in Informatics and Computing.

[28]  Simone Calderara,et al.  Understanding social relationships in egocentric vision , 2015, Pattern Recognit..

[29]  Alexandros André Chaaraoui,et al.  A review on vision techniques applied to Human Behaviour Analysis for Ambient-Assisted Living , 2012, Expert Syst. Appl..

[30]  Hamidur Rahman,et al.  Intelligent Driver Monitoring Based on Physiological Sensor Signals: Application Using Camera , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[31]  Kenji Mase,et al.  Activity and Location Recognition Using Wearable Sensors , 2002, IEEE Pervasive Comput..

[32]  HyungDal Park,et al.  A study on the activity classification using a triaxial accelerometer , 2003, Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE Cat. No.03CH37439).

[33]  Rita Cucchiara,et al.  People reidentification in surveillance and forensics , 2013, ACM Comput. Surv..

[34]  Sergio A. Velastin,et al.  Crowd analysis: a survey , 2008, Machine Vision and Applications.

[35]  Naokazu Yokoya,et al.  Networked video surveillance using multiple omnidirectional cameras , 2003, Proceedings 2003 IEEE International Symposium on Computational Intelligence in Robotics and Automation. Computational Intelligence in Robotics and Automation for the New Millennium (Cat. No.03EX694).

[36]  Mubarak Shah,et al.  A Lagrangian Particle Dynamics Approach for Crowd Flow Segmentation and Stability Analysis , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Guang-Zhong Yang,et al.  From Wearable Sensors to Smart Implants-–Toward Pervasive and Personalized Healthcare , 2015, IEEE Transactions on Biomedical Engineering.

[38]  Daniel McDuff,et al.  Improvements in Remote Cardiopulmonary Measurement Using a Five Band Digital Camera , 2014, IEEE Transactions on Biomedical Engineering.

[39]  Manuel P. Cuéllar,et al.  A survey on ontologies for human behavior recognition , 2014, ACM Comput. Surv..

[40]  Susan Elias,et al.  Probabilistic ontology based activity recognition in smart homes using Markov Logic Network , 2017, Knowl. Based Syst..

[41]  Ronald Poppe,et al.  A survey on vision-based human action recognition , 2010, Image Vis. Comput..

[42]  Li-Jia Li,et al.  Vision-Based Descriptive Analytics of Seniors ’ Daily Activities for Long-Term Health Monitoring , 2018 .

[43]  Rita Cucchiara,et al.  Real-time motion segmentation from moving cameras , 2004, Real Time Imaging.

[44]  Fabien Cardinaux,et al.  Video based technology for ambient assisted living: A review of the literature , 2011, J. Ambient Intell. Smart Environ..

[45]  Gianluigi Ferrari,et al.  Monitoring infants by automatic video processing: A unified approach to motion analysis , 2017, Comput. Biol. Medicine.

[46]  Zoran A. Salcic,et al.  Enhancing ontological reasoning with uncertainty handling for activity recognition , 2016, Knowl. Based Syst..

[47]  Jitendra Malik,et al.  Tracking people with twists and exponential maps , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[48]  Sufen Fong,et al.  MeshEye: A Hybrid-Resolution Smart Camera Mote for Applications in Distributed Intelligent Surveillance , 2007, 2007 6th International Symposium on Information Processing in Sensor Networks.

[49]  Marc Garbey,et al.  Contact-Free Measurement of Cardiac Pulse Based on the Analysis of Thermal Imagery , 2007, IEEE Transactions on Biomedical Engineering.

[50]  Angelo M. Sabatini,et al.  Machine Learning Methods for Classifying Human Physical Activity from On-Body Accelerometers , 2010, Sensors.

[51]  Dorin Comaniciu,et al.  Kernel-Based Object Tracking , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[52]  Venet Osmani,et al.  Human activity recognition in pervasive health-care: Supporting efficient remote collaboration , 2008, J. Netw. Comput. Appl..

[53]  Zoran A. Salcic,et al.  Adaptive sliding window segmentation for physical activity recognition using a single tri-axial accelerometer , 2017, Pervasive Mob. Comput..

[54]  Ilkka Korhonen,et al.  Detection of Daily Activities and Sports With Wearable Sensors in Controlled and Uncontrolled Conditions , 2008, IEEE Transactions on Information Technology in Biomedicine.

[55]  Ramin Mehran,et al.  Abnormal crowd behavior detection using social force model , 2009, CVPR.

[56]  Simon A. Dobson,et al.  Situation identification techniques in pervasive computing: A review , 2012, Pervasive Mob. Comput..

[57]  Rita Cucchiara,et al.  Probabilistic posture classification for Human-behavior analysis , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[58]  Tapio Seppänen,et al.  Recognizing human motion with multiple acceleration sensors , 2001, 2001 IEEE International Conference on Systems, Man and Cybernetics. e-Systems and e-Man for Cybernetics in Cyberspace (Cat.No.01CH37236).

[59]  Boris E. R. de Ruyter,et al.  SoPresent: An Awareness System for Connecting Remote Households , 2014, AmI.

[60]  Daniel McDuff,et al.  Advancements in Noncontact, Multiparameter Physiological Measurements Using a Webcam , 2011, IEEE Transactions on Biomedical Engineering.

[61]  Jake K. Aggarwal,et al.  Tracking Human Motion in Structured Environments Using a Distributed-Camera System , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[62]  Alexei A. Kamshilin,et al.  Photoplethysmographic imaging of high spatial resolution , 2011, Biomedical optics express.

[63]  István Z. Kovács,et al.  Interference Measurements in the European 868 MHz ISM Band with Focus on LoRa and SigFox , 2017, 2017 IEEE Wireless Communications and Networking Conference (WCNC).

[64]  Stephen J. McKenna,et al.  Activity summarisation and fall detection in a supportive home environment , 2004, ICPR 2004.

[65]  Miguel A. Labrador,et al.  A Survey on Human Activity Recognition using Wearable Sensors , 2013, IEEE Communications Surveys & Tutorials.

[66]  Yu Sun,et al.  Photoplethysmography Revisited: From Contact to Noncontact, From Point to Imaging , 2016, IEEE Transactions on Biomedical Engineering.

[67]  Zhiqi Shen,et al.  Physiological parameter monitoring of drivers based on video data and independent vector analysis , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[68]  Eliathamby Ambikairajah,et al.  Classification of a known sequence of motions and postures from accelerometry data using adapted Gaussian mixture models. , 2006, Physiological measurement.

[69]  Alessandro Perina,et al.  Person re-identification by symmetry-driven accumulation of local features , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[70]  Bernhard Rinner,et al.  Real-time video analysis on an embedded smart camera for traffic surveillance , 2004, Proceedings. RTAS 2004. 10th IEEE Real-Time and Embedded Technology and Applications Symposium, 2004..

[71]  James M. Keller,et al.  Monitoring patients in hospital beds using unobtrusive depth sensors , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[72]  R. Erts,et al.  The blood perfusion mapping in the human skin by photoplethysmography imaging , 2010 .

[73]  Tati L. R. Mengko,et al.  Personal medical assistant: Future exploration , 2011, Proceedings of the 2011 International Conference on Electrical Engineering and Informatics.

[74]  C. Diehl,et al.  Scheduling an active camera to observe people , 2004, VSSN '04.

[75]  Kamiar Aminian,et al.  Ambulatory system for human motion analysis using a kinematic sensor: monitoring of daily physical activity in the elderly , 2003, IEEE Transactions on Biomedical Engineering.

[76]  Mubarak Shah,et al.  Consistent Labeling of Tracked Objects in Multiple Cameras with Overlapping Fields of View , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[77]  Takeo Kanade,et al.  Introduction to the Special Section on Video Surveillance , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[78]  Noman Islam,et al.  A review of wireless sensors and networks' applications in agriculture , 2014, Comput. Stand. Interfaces.

[79]  Simone Calderara,et al.  Detecting anomalies in people's trajectories using spectral graph analysis , 2011, Comput. Vis. Image Underst..

[80]  Poonam J. Prasad Recent trend in wireless sensor network and its applications: a survey , 2015 .

[81]  Daniel McDuff,et al.  A survey of remote optical photoplethysmographic imaging methods , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[82]  Rosalind W. Picard,et al.  Automated Posture Analysis for Detecting Learner's Interest Level , 2003, 2003 Conference on Computer Vision and Pattern Recognition Workshop.

[83]  L. O. Svaasand,et al.  Remote plethysmographic imaging using ambient light. , 2008, Optics express.

[84]  T Togawa,et al.  Classification of waist-acceleration signals in a continuous walking record. , 2000, Medical engineering & physics.

[85]  Michael L. Littman,et al.  Activity Recognition from Accelerometer Data , 2005, AAAI.

[86]  Carlo Tomasi,et al.  Features for Multi-target Multi-camera Tracking and Re-identification , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[87]  Chris D. Nugent,et al.  An Ontology-Based Hybrid Approach to Activity Modeling for Smart Homes , 2014, IEEE Transactions on Human-Machine Systems.

[88]  J. Krumm,et al.  Multi-camera multi-person tracking for EasyLiving , 2000, Proceedings Third IEEE International Workshop on Visual Surveillance.

[89]  T. Ward,et al.  A CMOS Camera-Based Pulse Oximetry Imaging System , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[90]  Amitava Ghosh,et al.  NB-IoT system for M2M communication , 2016, 2016 IEEE Wireless Communications and Networking Conference.

[91]  Mingliang Chen,et al.  Spatiotemporal GMM for Background Subtraction with Superpixel Hierarchy , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[92]  Ahmed M. Elgammal Background Subtraction: Theory and Practice , 2014, Background Subtraction: Theory and Practice.

[93]  Greg Mori,et al.  Distinguishing the causes of falls in humans using an array of wearable tri-axial accelerometers. , 2014, Gait & posture.

[94]  Friedrich Foerster,et al.  Detection of posture and motion by accelerometry : a validation study in ambulatory monitoring , 1999 .

[95]  Lukas,et al.  On the application of IoT: Monitoring of troughs water level using WSN , 2015, 2015 IEEE Conference on Wireless Sensors (ICWiSe).

[96]  Sheng-Chieh Huang,et al.  A New Image Blood Pressure Sensor Based on PPG, RRT, BPTT, and Harmonic Balancing , 2014, IEEE Sensors Journal.

[97]  Sander Stuijk,et al.  Algorithmic Principles of Remote PPG , 2017, IEEE Transactions on Biomedical Engineering.

[98]  Geoff Mulligan,et al.  The 6LoWPAN architecture , 2007, EmNets '07.

[99]  J B Harness,et al.  Skin photoplethysmography--a review. , 1989, Computer methods and programs in biomedicine.

[100]  James M. Keller,et al.  Exploratory analysis of older adults' sedentary behavior in the primary living area using kinect depth data , 2017, J. Ambient Intell. Smart Environ..

[101]  Simone Calderara,et al.  Bayesian-Competitive Consistent Labeling for People Surveillance , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[102]  Konstantin Mikhaylov,et al.  Evaluation of LoRa LPWAN technology for remote health and wellbeing monitoring , 2016, 2016 10th International Symposium on Medical Information and Communication Technology (ISMICT).

[103]  Ian F. Akyildiz,et al.  Wireless sensor networks: a survey , 2002, Comput. Networks.

[104]  Cem Ersoy,et al.  Wireless sensor networks for healthcare: A survey , 2010, Comput. Networks.

[105]  Wendong Xiao,et al.  Daily Human Physical Activity Recognition Based on Kernel Discriminant Analysis and Extreme Learning Machine , 2015 .

[106]  Andrea Cavallaro,et al.  Multi-view Multi-object Detection and Tracking , 2010, Computer Vision: Detection, Recognition and Reconstruction.

[107]  Ashok Veeraraghavan,et al.  DistancePPG: Robust non-contact vital signs monitoring using a camera , 2015, Biomedical optics express.

[108]  Damian M. Lyons,et al.  Visual Surveillance in Retail Stores and in the Home , 2002 .

[109]  W. Eric L. Grimson,et al.  Adaptive background mixture models for real-time tracking , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[110]  Antonio Fernández-Caballero,et al.  A survey of video datasets for human action and activity recognition , 2013, Comput. Vis. Image Underst..

[111]  Caifeng Shan,et al.  Hybrid Optical Unobtrusive Blood Pressure Measurements , 2017, Sensors.

[112]  Aiguo Patrick Hu,et al.  Bipedal gait model for precise gait recognition and optimal triggering in foot drop stimulator: a proof of concept , 2018, Medical & Biological Engineering & Computing.

[113]  Michael D. Beynon,et al.  Detecting abandoned packages in a multi-camera video surveillance system , 2003, Proceedings of the IEEE Conference on Advanced Video and Signal Based Surveillance, 2003..

[114]  Kentaro Toyama,et al.  Wallflower: principles and practice of background maintenance , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[115]  Alexei A. Kamshilin,et al.  Variability of Microcirculation Detected by Blood Pulsation Imaging , 2013, PloS one.

[116]  Takeo Kanade,et al.  First-Person Vision , 2012, Proceedings of the IEEE.

[117]  Hironobu Fujiyoshi,et al.  Moving target classification and tracking from real-time video , 1998, Proceedings Fourth IEEE Workshop on Applications of Computer Vision. WACV'98 (Cat. No.98EX201).

[118]  Tim J. Ellis,et al.  Learning semantic scene models from observing activity in visual surveillance , 2005, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[119]  Gang Hua,et al.  Introduction to the Special Issue on Mobile Vision , 2011, International Journal of Computer Vision.

[120]  H. Busser,et al.  Ambulatory monitoring of physical activity in working situations, a validation study. , 1998, Journal of medical engineering & technology.

[121]  Carles Gomez,et al.  Wireless home automation networks: A survey of architectures and technologies , 2010, IEEE Communications Magazine.

[122]  Jean Meunier,et al.  Video Surveillance for Fall Detection , 2011 .

[123]  Sijung Hu,et al.  Feasibility of Imaging Photoplethysmography , 2008, 2008 International Conference on BioMedical Engineering and Informatics.

[124]  Stefano Chessa,et al.  Wireless sensor networks: A survey on the state of the art and the 802.15.4 and ZigBee standards , 2007, Comput. Commun..

[125]  Sijung Hu,et al.  The preliminary investigation of imaging photoplethysmographic system , 2007 .

[126]  Kevin I-Kai Wang,et al.  Ontology-based sensor fusion activity recognition , 2020, J. Ambient Intell. Humaniz. Comput..

[127]  Larry S. Davis,et al.  Multi-camera Tracking and Segmentation of Occluded People on Ground Plane Using Search-Guided Particle Filtering , 2006, ECCV.

[128]  Andrea Prati,et al.  Integrating Consumer Smart Cameras into Camera Networks: Opportunities and Obstacles , 2014, Computer.

[129]  Alessio Del Bue,et al.  Social interaction discovery by statistical analysis of F-formations , 2011, BMVC.

[130]  James W. Davis,et al.  Real-time recognition of activity using temporal templates , 1996, Proceedings Third IEEE Workshop on Applications of Computer Vision. WACV'96.

[131]  Seung-Hoon Hwang,et al.  A survey on LPWA technology: LoRa and NB-IoT , 2017, ICT Express.

[132]  Magdalena Madej,et al.  Measuring Pulse Rate with a Webcam , 2012 .

[133]  Simone Calderara,et al.  Visual Tracking: An Experimental Survey , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[134]  Abdenour Hadid,et al.  A Survey on Computer Vision for Assistive Medical Diagnosis From Faces , 2018, IEEE Journal of Biomedical and Health Informatics.

[135]  Marcello Pelillo,et al.  Multi-object tracking using dominant sets , 2016, IET Comput. Vis..

[136]  W. Eric L. Grimson,et al.  Learning Patterns of Activity Using Real-Time Tracking , 2000, IEEE Trans. Pattern Anal. Mach. Intell..