Development of Home Intelligent Fall Detection IoT System Based on Feedback Optical Flow Convolutional Neural Network

Fall events are important health issues in elderly living environments such as homes. Hence, a confident and real-time video surveillance device that pays attention could better their everyday lives. We proposed an optical flow feedback convolutional neural network according to the video stream in a home environment. Our proposed model uses rule-based filters before an input convolutional layer and the recorded optical flow for supervising the optical flow of variation. Detecting human posture is a key factor, while fall events are like a falling posture. By sequencing frames of action, it is possible to recognize a fall. Our system can clearly detect the normal lying posture and lying after falling. Our proposed method can efficiently detect action motion and recognize the action posture. We compared the performance with other standard benchmark data sets and deployed our model to simulate a real-home situation, and the correct ratio achieved 82.7% and 98% separately.

[1]  Takuya Maekawa,et al.  Activity recognition with hand-worn magnetic sensors , 2013, Personal and Ubiquitous Computing.

[2]  Kamiar Aminian,et al.  Gait analysis using shoe-worn inertial sensors: how is foot clearance related to walking speed? , 2014, UbiComp.

[3]  Ran El-Yaniv,et al.  Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations , 2016, J. Mach. Learn. Res..

[4]  C. Lawrence Zitnick,et al.  Edge Boxes: Locating Object Proposals from Edges , 2014, ECCV.

[5]  Qiang Chen,et al.  Design of a terminal solution for integration of in-home health care devices and services towards the Internet-of-Things , 2015, Enterp. Inf. Syst..

[6]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Yingli Tian,et al.  Privacy Preserving Automatic Fall Detection for Elderly Using RGBD Cameras , 2012, ICCHP.

[8]  Frédérique C. Pivot,et al.  Fall Detection and Prevention for the Elderly: A Review of Trends and Challenges , 2013 .

[9]  Didier Dubois,et al.  Possibility Theory - An Approach to Computerized Processing of Uncertainty , 1988 .

[10]  Diane J. Cook,et al.  Human Activity Recognition and Pattern Discovery , 2010, IEEE Pervasive Computing.

[11]  Pearl Pu,et al.  Predictors of life satisfaction based on daily activities from mobile sensor data , 2014, CHI.

[12]  Mark W. Newman,et al.  The Work to Make a Home Network Work , 2005, ECSCW.

[13]  Shuangquan Wang,et al.  A Nonintrusive and Single-Point Infrastructure-Mediated Sensing Approach for Water-Use Activity Recognition , 2013, 2013 IEEE 10th International Conference on High Performance Computing and Communications & 2013 IEEE International Conference on Embedded and Ubiquitous Computing.

[14]  Luc Van Gool,et al.  Action snippets: How many frames does human action recognition require? , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Bogdan Kwolek,et al.  Unobtrusive Fall Detection at Home Using Kinect Sensor , 2013, CAIP.

[16]  Anthony Rowe,et al.  MARS: A Muscle Activity Recognition System enabling self-configuring musculoskeletal sensor networks , 2013, 2013 ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN).

[17]  Marjorie Skubic,et al.  Doppler Radar Fall Activity Detection Using the Wavelet Transform , 2015, IEEE Transactions on Biomedical Engineering.

[18]  Emiliano Miluzzo,et al.  CenceMe - Injecting Sensing Presence into Social Networking Applications , 2007, EuroSSC.

[19]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Martin L. Griss,et al.  NuActiv: recognizing unseen new activities using semantic attribute-based learning , 2013, MobiSys '13.

[21]  Archan Misra,et al.  Managing location in "Universal" location-aware computing , 2003 .

[22]  Dieter Fox,et al.  A Scalable Tree-Based Approach for Joint Object and Pose Recognition , 2011, AAAI.

[23]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[24]  Emile A. Hendriks,et al.  Markerless human motion capture and pose recognition , 2009, 2009 10th Workshop on Image Analysis for Multimedia Interactive Services.

[25]  Kent Larson,et al.  A living laboratory for the design and evaluation of ubiquitous computing technologies , 2005, CHI Extended Abstracts.

[26]  Ramin Mehran,et al.  Abnormal crowd behavior detection using social force model , 2009, CVPR.

[27]  Victor R. L. Shen,et al.  The implementation of a smartphone-based fall detection system using a high-level fuzzy Petri net , 2015, Appl. Soft Comput..

[28]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[29]  Shuangquan Wang,et al.  PPCare: A Personal and Pervasive Health Care System for the Elderly , 2012, 2012 9th International Conference on Ubiquitous Intelligence and Computing and 9th International Conference on Autonomic and Trusted Computing.

[30]  Israel Gannot,et al.  A Method for Automatic Fall Detection of Elderly People Using Floor Vibrations and Sound—Proof of Concept on Human Mimicking Doll Falls , 2009, IEEE Transactions on Biomedical Engineering.

[31]  Xiang Chen,et al.  A Framework for Daily Activity Monitoring and Fall Detection Based on Surface Electromyography and Accelerometer Signals , 2013, IEEE Journal of Biomedical and Health Informatics.

[32]  Huosheng Hu,et al.  Myoelectric control systems - A survey , 2007, Biomed. Signal Process. Control..

[33]  Guoliang Xing,et al.  iSleep: unobtrusive sleep quality monitoring using smartphones , 2013, SenSys '13.

[34]  Gang Zhou,et al.  Accurate, fast fall detection using posture and context information , 2008, SenSys '08.

[35]  Gwenn Englebienne,et al.  An activity monitoring system for elderly care using generative and discriminative models , 2010, Personal and Ubiquitous Computing.

[36]  Antonis A. Argyros,et al.  Efficient model-based 3D tracking of hand articulations using Kinect , 2011, BMVC.

[37]  Martial Hebert,et al.  Spatio-temporal Shape and Flow Correlation for Action Recognition , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[38]  John Nelson,et al.  DESIGN AND TEST OF A LONG-TERM FALL DETECTION SYSTEM INCORPORATED INTO A CUSTOM VEST FOR THE ELDERLY. , 2008 .

[39]  Manuel Prado-Velasco,et al.  The inescapable smart impact detection system (ISIS): An ubiquitous and personalized fall detector based on a distributed “divide and conquer strategy” , 2008, 2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[40]  Nasser Kehtarnavaz,et al.  Inertial Measurement Unit-Based Wearable Computers for Assisted Living Applications: A signal processing perspective , 2016, IEEE Signal Processing Magazine.

[41]  Guoliang Xing,et al.  Remora: Sensing resource sharing among smartphone-based body sensor networks , 2013, 2013 IEEE/ACM 21st International Symposium on Quality of Service (IWQoS).

[42]  Jenq-Neng Hwang,et al.  A Review on Video-Based Human Activity Recognition , 2013, Comput..

[43]  Dieter Fox,et al.  Fine-grained kitchen activity recognition using RGB-D , 2012, UbiComp.

[44]  Tom Rodden,et al.  At home with the technology: an ethnographic study of a set-top-box trial , 1999, TCHI.

[45]  Didier Dubois,et al.  A possibility theory-based approach to the handling of uncertain relations between temporal points , 2004, Proceedings. 11th International Symposium on Temporal Representation and Reasoning, 2004. TIME 2004..

[46]  Shuwan Xue,et al.  Portable Preimpact Fall Detector With Inertial Sensors , 2008, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[47]  Amir Davari,et al.  Automatic fall detection for elderly by using features extracted from skeletal data , 2013, 2013 International Conference on Electronics, Computer and Computation (ICECCO).

[48]  A. Bourke,et al.  Fall detection - Principles and Methods , 2007, 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[49]  David Wetherall,et al.  Recognizing daily activities with RFID-based sensors , 2009, UbiComp.

[50]  Svetha Venkatesh,et al.  Activity recognition and abnormality detection with the switching hidden semi-Markov model , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[51]  Diane J. Cook,et al.  Mining Sensor Streams for Discovering Human Activity Patterns over Time , 2010, 2010 IEEE International Conference on Data Mining.

[52]  Gang Zhou,et al.  Accurate, Fast Fall Detection Using Gyroscopes and Accelerometer-Derived Posture Information , 2009, 2009 Sixth International Workshop on Wearable and Implantable Body Sensor Networks.

[53]  Jun Zhang,et al.  Image-Based Fall Detection with Human Posture Sequence Modeling , 2013, 2013 IEEE International Conference on Healthcare Informatics.

[54]  Yaser Mowafi,et al.  Fall detection for elderly using anatomical-plane-based representation , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[55]  Alex Pentland,et al.  Pfinder: real-time tracking of the human body , 1996, Proceedings of the Second International Conference on Automatic Face and Gesture Recognition.

[56]  Ennio Gambi,et al.  Radar and RGB-Depth Sensors for Fall Detection: A Review , 2017, IEEE Sensors Journal.

[57]  Serge J. Belongie,et al.  Behavior recognition via sparse spatio-temporal features , 2005, 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance.

[58]  Norbert Wehn,et al.  Monitoring household activities and user location with a cheap, unobtrusive thermal sensor array , 2014, UbiComp.

[59]  Roger Orpwood,et al.  The design of smart homes for people with dementia—user-interface aspects , 2005, Universal Access in the Information Society.

[60]  Guoliang Xing,et al.  PBN: towards practical activity recognition using smartphone-based body sensor networks , 2011, SenSys.

[61]  M. Skubic,et al.  Older adults' attitudes towards and perceptions of ‘smart home’ technologies: a pilot study , 2004, Medical informatics and the Internet in medicine.

[62]  Barbara Caputo,et al.  Recognizing human actions: a local SVM approach , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[63]  Alexandra König,et al.  Validation of an automatic video monitoring system for the detection of instrumental activities of daily living in dementia patients. , 2015, Journal of Alzheimer's disease : JAD.

[64]  Chung-Lin Huang,et al.  Slip and fall event detection using Bayesian Belief Network , 2012, Pattern Recognit..

[65]  Chris D. Nugent,et al.  Home Based Assistive Technologies for People with Mild Dementia , 2007, ICOST.

[66]  Xiang Zhang,et al.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks , 2013, ICLR.

[67]  Huosheng Hu,et al.  New Fast Fall Detection Method Based on Spatio-Temporal Context Tracking of Head by Using Depth Images , 2015, Sensors.

[68]  James J. Little,et al.  Simultaneous Tracking and Action Recognition using the PCA-HOG Descriptor , 2006, The 3rd Canadian Conference on Computer and Robot Vision (CRV'06).

[69]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[70]  Miguel Terroso,et al.  Physical consequences of falls in the elderly: a literature review from 1995 to 2010 , 2014, European Review of Aging and Physical Activity.

[71]  Vassilis Athitsos,et al.  A survey on vision-based fall detection , 2015, PETRA.

[72]  Emmanuel Andrès,et al.  From Fall Detection to Fall Prevention: A Generic Classification of Fall-Related Systems , 2017, IEEE Sensors Journal.

[73]  Juan Carlos Niebles,et al.  Unsupervised Learning of Human Action Categories , 2006 .

[74]  Allison Druin,et al.  Technology probes: inspiring design for and with families , 2003, CHI '03.

[75]  Rui Liu,et al.  Fall detection for elderly person care in a vision-based home surveillance environment using a monocular camera , 2014, Signal Image Video Process..

[76]  Koen E. A. van de Sande,et al.  Selective Search for Object Recognition , 2013, International Journal of Computer Vision.

[77]  Yiqiang Chen,et al.  RECOGNIZING TRANSPORTATION MODE ON MOBILE PHONE USING PROBABILITY FUSION OF EXTREME LEARNING MACHINES , 2013 .

[78]  John Cosmas,et al.  A comparison between heuristic and machine learning techniques in fall detection using Kinect v2 , 2016, 2016 IEEE International Symposium on Medical Measurements and Applications (MeMeA).

[79]  Donald A. Norman Home Theater: Not Ready for Prime Time , 2002, Computer.

[80]  Amy Loutfi,et al.  Challenges and Issues in Multisensor Fusion Approach for Fall Detection: Review Paper , 2016, J. Sensors.

[81]  Christopher Joseph Pal,et al.  Activity recognition using the velocity histories of tracked keypoints , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[82]  David A. McAllester,et al.  Object Detection with Discriminatively Trained Part Based Models , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[83]  Shir-Kuan Lin,et al.  Fall detection for multiple pedestrians using depth image processing technique , 2014, Comput. Methods Programs Biomed..

[84]  Mark Weiser,et al.  The computer for the 21st Century , 1991, IEEE Pervasive Computing.

[85]  Ling Shao,et al.  A survey on fall detection: Principles and approaches , 2013, Neurocomputing.

[86]  Shuangquan Wang,et al.  b-COELM: A fast, lightweight and accurate activity recognition model for mini-wearable devices , 2014, Pervasive Mob. Comput..

[87]  Shahram Izadi,et al.  ECT: A Toolkit to Support Rapid Construction of Ubicomp Environments , 2004 .

[88]  Bill N. Schilit,et al.  Place Lab: Device Positioning Using Radio Beacons in the Wild , 2005, Pervasive.

[89]  James H. Aylor,et al.  Computer for the 21st Century , 1999, Computer.

[90]  Thomas Serre,et al.  A Biologically Inspired System for Action Recognition , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[91]  Björn Eskofier,et al.  Classification of Daily Life Activities by Decision Level Fusion of Inertial Sensor Data , 2013, BODYNETS.

[92]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[93]  Miguel A. Labrador,et al.  A Survey on Human Activity Recognition using Wearable Sensors , 2013, IEEE Communications Surveys & Tutorials.

[94]  Ming Yang,et al.  3D Convolutional Neural Networks for Human Action Recognition , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[95]  Yanjun Jia,et al.  Diatetic and Exercise Therapy against Diabetes Mellitus , 2009, 2009 Second International Conference on Intelligent Networks and Intelligent Systems.

[96]  Qiang Yang,et al.  Sensor-Based Abnormal Human-Activity Detection , 2008, IEEE Transactions on Knowledge and Data Engineering.

[97]  Xiaofeng Ren,et al.  Figure-ground segmentation improves handled object recognition in egocentric video , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[98]  Matthew Keally,et al.  AdaSense: Adapting sampling rates for activity recognition in Body Sensor Networks , 2013, 2013 IEEE 19th Real-Time and Embedded Technology and Applications Symposium (RTAS).